50
PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN EVALUATION OF THE MILWAUKEE PARENTAL CHOICE PROGRAM* CECILIA ELENA ROUSE In 1990 Wisconsin began providing vouchers to a small number of low-income students to attend nonsectarian private schools. Controlling for individual xed- effects, I compare the test scores of students selected to attend a participating private school with those of unsuccessful applicants and other students from the Milwaukee public schools. I nd that students in the Milwaukee Parental Choice Program had faster math score gains than, but similar reading score gains to, the comparison groups. The results appear robust to data imputations and sample attrition, although these de ciencies of the data should be kept in mind when interpreting the results. I. INTRODUCTION At the cornerstone of many school reform proposals lies the premise that private schools are more efficient than public schools. This premise is particularly prominent in the current ‘‘school choice’’ debate. Proponents of school choice argue that governments should offer tuition vouchers to families who wish to send their children to private, rather than public, schools. If private schools are indeed more effective than public schools, a voucher program may offer a cost-effective way to improve the quality of education. The original evidence of more effective private schooling is a study by Coleman, Hoffer, and Kilgore {1982a, 1982b} who used the rst year of the High School and Beyond data to show that students in private schools have higher achievement levels than those in public schools. Critics of school choice programs have argued that private schools would not necessarily do a better job educating students who are currently attending public schools. Rather, they argue that the observed superiority of private school students arises from the selection * I thank Orley Ashenfelter, Michael Boozer, Kristin Butcher, David Card, Henry Farber, and Alan Krueger for insightful suggestions and conversations, participants at the Demand-side Financing in Education seminar at the World Bank, the University of Chicago Business School’s labor seminar, the National Bureau of Economic Research’s Program on Children Conference, the Princeton Labor Lunch, and the University of Wisconsin at Madison’s public nance seminar, three anonymous referees, and the editor, Lawrence Katz, for helpful comments. I particularly thank Lisa Boeger, Jay Greene, and John Witte for insights about the program and lots of help with the data. Jeffrey Wilder provided expert research assistance. I thank the Mellon Foundation and the Center for Economic Policy Studies at Princeton University for nancial support. All errors are mine. r 1998 by the President and Fellows of Harvard College and the Massachusetts Institute of Technology. The Quarterly Journal of Economics, May 1998

PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

PRIVATE SCHOOL VOUCHERS AND STUDENTACHIEVEMENT AN EVALUATION OF THE MILWAUKEE

PARENTAL CHOICE PROGRAM

CECILIA ELENA ROUSE

In 1990 Wisconsin began providing vouchers to a small number of low-incomestudents to attend nonsectarian private schools Controlling for individual xed-effects I compare the test scores of students selected to attend a participatingprivate school with those of unsuccessful applicants and other students from theMilwaukee public schools I nd that students in the Milwaukee Parental ChoiceProgram had faster math score gains than but similar reading score gains to thecomparison groups The results appear robust to data imputations and sampleattrition although these deciencies of the data should be kept in mind wheninterpreting the results

I INTRODUCTION

At the cornerstone of many school reform proposals lies thepremise that private schools are more efficient than publicschools This premise is particularly prominent in the currentlsquolsquoschool choicersquorsquo debate Proponents of school choice argue thatgovernments should offer tuition vouchers to families who wish tosend their children to private rather than public schools Ifprivate schools are indeed more effective than public schools avoucher program may offer a cost-effective way to improve thequality of education The original evidence of more effectiveprivate schooling is a study by Coleman Hoffer and Kilgore1982a 1982b who used the rst year of the High School andBeyond data to show that students in private schools have higherachievement levels than those in public schools Critics of schoolchoice programs have argued that private schools would notnecessarily do a better job educating students who are currentlyattending public schools Rather they argue that the observedsuperiority of private school students arises from the selection

I thank Orley Ashenfelter Michael Boozer Kristin Butcher David CardHenry Farber and Alan Krueger for insightful suggestions and conversationsparticipants at the Demand-side Financing in Education seminar at the WorldBank the University of Chicago Business Schoolrsquos labor seminar the NationalBureau of Economic Researchrsquos Program on Children Conference the PrincetonLabor Lunch and the University of Wisconsin at Madisonrsquos public nance seminarthree anonymous referees and the editor Lawrence Katz for helpful comments Iparticularly thank Lisa Boeger Jay Greene and John Witte for insights about theprogram and lots of help with the data Jeffrey Wilder provided expert researchassistance I thank the Mellon Foundation and the Center for Economic PolicyStudies at Princeton University for nancial support All errors are mine

r 1998 by the President and Fellows of Harvard College and the Massachusetts Institute ofTechnologyThe Quarterly Journal of Economics May 1998

process that leads higher-achieving students to attend privateschools Goldberger and Cain 1982 Cain and Goldgerger 19831

Ideally the issue of the relative effectiveness of private versuspublic schooling could be addressed by a social experiment inwhich children in a well-dened universe were randomly assignedto a private school (the lsquolsquotreatment grouprsquorsquo) while others wereassigned to attend public schools (the lsquolsquocontrol grouprsquorsquo) After someperiod of time one could compare outcomes such as test scoreshigh school graduation rates or labor market success between thetreatment and control groups Since on average the only differ-ences between the groups would be their initial assignmentmdashwhich was randomly determinedmdashany differences in outcomescould be attributed to the type of school attended While such anexperiment has never been implemented the legislative require-ments of a recently enacted school voucher program in Milwau-kee Wisconsin theoretically allow one to come close to such anidealized experiment

In 1990 Wisconsin became the rst state in the country toimplement a school choice program that provides vouchers tolow-income students to attend nonsectarian private schools2 Thenumber of students in any year was originally limited to 1 percentof the Milwaukee public schools membership but was expandedto 15 percent in 1994 Only students whose family income was ator below 175 times the national poverty line were eligible toapply In principle a child from a family of three with an income ofapproximately $21000 was eligible to apply to the program inpractice as shown in Table I the mean family income of appli-cants was approximately $12300 The choice students wereconsiderably more disadvantaged than the average student in theMilwaukee public schools (whose average family income was$24000) and the average nonchoice private school student inMilwaukee (whose average family income was about $43000according to Witte Thorn and Pritchard 1995) The means inTable I also show that choice applicants were more likely to beminority and had lower math and reading test scores than theaverage student in the Milwaukee public schools These test

1 See as well the collection of articles in the Sociology of Education 1982and the Harvard Education Review 1981 as well as the review pieces by Cookson1993 Murnane 1984 and Witte 1992

2 There was an attempt to include religious schools in the program Howeverthe Wisconsin Supreme Court ruled that it would violate the separation betweenchurch and state I only briey describe the program here For more details seeWitte Thorn Pritchard and Claibourn 1994 and Witte Sterr and Thorn 1995

QUARTERLY JOURNAL OF ECONOMICS554

TABLE IMEAN CHARACTERISTICS OF APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS IN

THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee

public schoolssampleSelected

Not-selected

African-American 0721 0780 0550(0011) (0012) (0007)

Hispanic 0217 0154 0097(0011) (0011) (0004)

lsquolsquoFirstrsquorsquo math score (national percentileranking)

34560(1196)

35073(1552)

43353(0494)

Proportion missing lsquolsquorstrsquorsquo math testscore

0704(0012)

0779(0012)

0382(0007)

lsquolsquoFirstrsquorsquo reading score (national percen-tile ranking)

33869(1050)

34341(1369)

40227(0462)

Proportion missing lsquolsquorstrsquorsquo readingscore

0699(0012)

0781(0012)

0382(0007)

Family income ( 4 1000) (1994 dollars) $12123(0297)

$12715(0509)

$23897(0534)

Proportion missing family income 0491(0013)

0760(0012)

0727(0006)

Motherrsquos education 12559 12399 12108(0067) (0109) (0055)

Proportion missing motherrsquos education 0555(0013)

0763(0012)

0724(0006)

Fatherrsquos education 12048 11521 12250(0092) (0152) (0068)

Proportion missing fatherrsquos education 0697(0012)

0846(0010)

0796(0005)

Grade of application 2997 3875 3481(0079) (0119) (0058)

Proportion with math test score twoyears after application

0536(0015)

0284(0019)

0535(0009)

Proportion with math test score threeyears after application

0467(0017)

0236(0024)

0405(0009)

Proportion with math test score fouryears after application

0407(0025)

0183(0027)

0364(0011)

Maximum number of observations 1544 1219 5318

Standard errors are in parentheses For applicants to the choice schools the lsquolsquorstrsquorsquo test score is thelsquolsquopreapplication test scorersquorsquo (the test score from the year of application) or an earlier one if the application testscore is missing for students in the Milwaukee public schools the lsquolsquorstrsquorsquo test score is the rst nonmissing(post-1989) test score See the Data Appendix or Section III for a description of the test scores See the DataAppendix for how I construct a lsquolsquoyear of applicationrsquorsquo for the Milwaukee public schools sample This sample onlyincludes one observation per student and is based on all available data (it does not represent my analysissample)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 555

scores are nationally normed suggesting that the students whoapplied to choice were scoring considerably lower than the na-tional average as well3 On the other hand the parental educa-tion for choice applicants at least for those responding to thesurveys was comparable to (or even a little higher than) nonappli-cants from the Milwaukee public schools

As the program limited participation to independent secularprivate schools the participating schools were not representativeof all private schools in Milwaukee where the majority of privateschool enrollments are likely in religious schools Witte Thornand Pritchard 1995 That said until the constitutionality ofwhether religious schools can participate in voucher programs hasbeen decided the experience in Milwaukee will be relevant forother cities considering such reforms In the rst year sevenprivate schools participated by 1995 this number had risen totwelve These schools represented a variety of educational ap-proaches including Montessori Waldorf bilingual and African-American cultural emphases These were not elite private schoolsFor example the voucher was worth approximately $3200 in1994ndash1995 which contrasts with a range of tuition and fees forschools participating in the choice program of $1080ndash$4000 (in1993ndash1994) In fact the voucher program helped to improve thenancial status of several of the participating schools While onecould believe a priori that a program providing vouchers to eliteprivate schools would result in higher achievement gains it is notso clear that a program providing vouchers to local nonsectarianprivate schools (that are willing to participate) would have sucheffects

Finally although in the Milwaukee Parental Choice Programstate aid followed the students from the public schools to theprivate schools the choice program was too small to provideinsight into the potential general equilibrium student achieve-ment benets of large-scale lsquolsquoschool choicersquorsquo4 In the most unre-stricted school choice program all (or a substantial fraction) of thestudents in the public schools would be eligible to attend a privateschool Since state funding would be tied to student enrollmentsthe public schools would have an incentive to improve leading tono differences in the outcomes of students in lsquolsquopublicrsquorsquo and lsquolsquoprivatersquorsquo

3 See the Data Appendix or Section III for a description of the test scores4 In addition choice schools were only allowed to admit up to 49 percent of

their students as part of the choice program (this level was raised to 65 percent in1994) which limited any potential supply response by private schools

QUARTERLY JOURNAL OF ECONOMICS556

schools in the long run5 Analysis of the Milwaukee ParentalChoice Program can however indicate whether parents wouldprefer to send their children to a private school (see Witte Sterrand Thorn 1995) and whether in the short run the academicachievement of those children who are selected for the programand those who attend the private schools would likely increase

The original evaluation of the fourth year of the choiceprogram conducted by Witte Sterr and Thorn 1995 comparesthe test scores of students in the choice schools with those of arandom sample of all Milwaukee public schools students andwith low-income students and concludes that there were nostatistically signicant relative achievement gains among thechoice students (see also Witte 1997) A subsequent analysis byGreene Peterson Du Boeger and Frazier 1996 criticizes theWitte Sterr and Thorn study for using a comparison group thatwas from substantially more advantaged families than studentsin the choice program Although Witte Sterr and Thorn arguethat their comparison group is in fact comparable to choicestudents conditional on observable covariates it is neverthelesspossible that unobserved factors remain which would tend toobscure any relative achievement gains among the choice studentpopulation

As an alternative to the use of a general comparison groupGreene et al 1996 propose the use of the unsuccessful applicantsas a lsquolsquoquasi-experimentalrsquorsquo control group Relative to the unsuccess-ful applicants Greene et al conclude that the choice studentsmade statistically signicant test score gains by their third andfourth years in the program in both reading and math Howeverthe analysis by Greene et al may overstate the effect of theprogram by excluding from the choice group students who weresuccessfully admitted to the choice schools but did not attendthem or attended only for a short period of time In addition theunsuccessful applicants may not provide an ideal control groupsince those who remained in the Milwaukee public schools appearto have been a nonrandom subset of all unsuccessful applicants

In this paper I use both the unsuccessful applicants and therandom sample of students from the Milwaukee public schools ascomparison groups In addition I return to the lsquolsquotruersquorsquo source ofexogenous variation in the Milwaukee Choice programmdashthat of

5 See Epple and Romano 1996 and Nechyba 1996 for theoretical models ofthe political economy and achievement effects of school vouchers and Hoxby 1996for an (indirect) empirical study

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 557

selection into the pool of students eligible to attend a choiceschool I argue that a complete evaluation of the programrsquos effectson student achievement should consider not only whether privateschools are better than public schools (as do Greene et al 19961997 Witte Sterr and Thorn 1995 and Witte 1997) but alsowhether students who were selected to attend a choice schoolenrolled and remained there I do so by estimating the effect ofbeing selected to attend a choice school on student achievementBecause the unsuccessful applicants are a potentially problematiccontrol group I also compare the test score gains of studentsselected to the choice program with the gains of a random sampleof students in the Milwaukee public schools To control fortime-invariant differences between students selected for theprogram and the comparison groups I implement an individualxed-effects strategy Finally I estimated structural equations ofthe relative effectiveness of the choice schools and the publicschools in Milwaukee

I nd that students selected for the choice program scoredapproximately 15ndash23 extra percentile points per year in mathcompared with unsuccessful applicants and the sample of otherstudents in the Milwaukee public schools The achievement gainsof those actually enrolled in the choice schools were quite similarGiven a (within-sample) standard deviation of about nineteenpercentile points on the math test this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions I do notestimate statistically signicant differences between sectors inreading scores

Some have argued that the key difference between theexisting analyses by Greene et al and Witte et al is thecontrolcomparison group However I nd that the two compari-son groups yield similar estimates when individual xed-effectsare included although the results using the unsuccessful appli-cants are not as robust to variation in sample My results on theeffects on reading scores differ from those reported by Greene etal because their reading results are not robust to the inclusion ofindividual xed-effects and to alternative specications My re-sults for math differ from those reported by Witte partly becauseof our specications and partly because my xed-effects specica-tion takes advantage of a larger sample

Finally these data are not ideal and the problems threatenthe validity of any evaluation of the Milwaukee Parental Choice

QUARTERLY JOURNAL OF ECONOMICS558

Program I have found my results to be robust to substantialmissing and imputed data and (potentially) nonrandom selectionand attrition from the sample Nevertheless because econometrictechniques cannot substitute for better data these data decien-cies should be kept in mind when interpreting the results

II EMPIRICAL STRATEGIES

A Using the Unsuccessful Applicants as a Control Group

In an lsquolsquoidealrsquorsquo experiment the random assignment processcompletely determines the status of individuals in the treatmentand control groups In many experimental settings involvinghuman subjects however there is slippage between the randomassignment status of experimental subjects and whether or notthey actually receive the treatment For example in randomizedtrials of medicines some individuals in the treatment group mayfail to take the prescribed treatment (see for example CoronaryDrug Project 1980 and Efron and Feldman 1991) In somesocial experiments the slippage can occur in both directions Forexample in a job training experiment some people assigned tothe treatment group may fail to show up whereas others in thecontrol group may receive training from other providers Haus-man and Wise 1985 Heckman and Smith 1994 In the speciccontext of the Milwaukee Choice program the slippage betweenrandom assignment statusmdashwhether the individual was chosento attend a choice schoolmdashand lsquolsquotreatment statusrsquorsquomdashwhether anindividual actually attended a choice schoolmdashis signicant be-cause the treatment lasted over several years In fact only about50 percent of students who were selected to attend choice schoolsin the rst year of the program were still attending the school twoyears later In order to understand the nature of alternativeestimates of the relative effect of the choice program in thissetting it is necessary to develop a model that takes account of thelink between the random selection process and the decision toattend a choice school

Consider the following linear probability model that modelschild irsquos actual attendance at a choice school in year t Pit (Pit 5 1for students who attend a choice school Pit 5 0 for those who donot) as a function of irsquos selection status in the previous year t 2 1Sit 2 1 (Sit 2 1 5 1 for students who were selected in year t 2 1Sit 2 1 5 0 for those who were either never selected or were selected

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 559

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 2: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

process that leads higher-achieving students to attend privateschools Goldberger and Cain 1982 Cain and Goldgerger 19831

Ideally the issue of the relative effectiveness of private versuspublic schooling could be addressed by a social experiment inwhich children in a well-dened universe were randomly assignedto a private school (the lsquolsquotreatment grouprsquorsquo) while others wereassigned to attend public schools (the lsquolsquocontrol grouprsquorsquo) After someperiod of time one could compare outcomes such as test scoreshigh school graduation rates or labor market success between thetreatment and control groups Since on average the only differ-ences between the groups would be their initial assignmentmdashwhich was randomly determinedmdashany differences in outcomescould be attributed to the type of school attended While such anexperiment has never been implemented the legislative require-ments of a recently enacted school voucher program in Milwau-kee Wisconsin theoretically allow one to come close to such anidealized experiment

In 1990 Wisconsin became the rst state in the country toimplement a school choice program that provides vouchers tolow-income students to attend nonsectarian private schools2 Thenumber of students in any year was originally limited to 1 percentof the Milwaukee public schools membership but was expandedto 15 percent in 1994 Only students whose family income was ator below 175 times the national poverty line were eligible toapply In principle a child from a family of three with an income ofapproximately $21000 was eligible to apply to the program inpractice as shown in Table I the mean family income of appli-cants was approximately $12300 The choice students wereconsiderably more disadvantaged than the average student in theMilwaukee public schools (whose average family income was$24000) and the average nonchoice private school student inMilwaukee (whose average family income was about $43000according to Witte Thorn and Pritchard 1995) The means inTable I also show that choice applicants were more likely to beminority and had lower math and reading test scores than theaverage student in the Milwaukee public schools These test

1 See as well the collection of articles in the Sociology of Education 1982and the Harvard Education Review 1981 as well as the review pieces by Cookson1993 Murnane 1984 and Witte 1992

2 There was an attempt to include religious schools in the program Howeverthe Wisconsin Supreme Court ruled that it would violate the separation betweenchurch and state I only briey describe the program here For more details seeWitte Thorn Pritchard and Claibourn 1994 and Witte Sterr and Thorn 1995

QUARTERLY JOURNAL OF ECONOMICS554

TABLE IMEAN CHARACTERISTICS OF APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS IN

THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee

public schoolssampleSelected

Not-selected

African-American 0721 0780 0550(0011) (0012) (0007)

Hispanic 0217 0154 0097(0011) (0011) (0004)

lsquolsquoFirstrsquorsquo math score (national percentileranking)

34560(1196)

35073(1552)

43353(0494)

Proportion missing lsquolsquorstrsquorsquo math testscore

0704(0012)

0779(0012)

0382(0007)

lsquolsquoFirstrsquorsquo reading score (national percen-tile ranking)

33869(1050)

34341(1369)

40227(0462)

Proportion missing lsquolsquorstrsquorsquo readingscore

0699(0012)

0781(0012)

0382(0007)

Family income ( 4 1000) (1994 dollars) $12123(0297)

$12715(0509)

$23897(0534)

Proportion missing family income 0491(0013)

0760(0012)

0727(0006)

Motherrsquos education 12559 12399 12108(0067) (0109) (0055)

Proportion missing motherrsquos education 0555(0013)

0763(0012)

0724(0006)

Fatherrsquos education 12048 11521 12250(0092) (0152) (0068)

Proportion missing fatherrsquos education 0697(0012)

0846(0010)

0796(0005)

Grade of application 2997 3875 3481(0079) (0119) (0058)

Proportion with math test score twoyears after application

0536(0015)

0284(0019)

0535(0009)

Proportion with math test score threeyears after application

0467(0017)

0236(0024)

0405(0009)

Proportion with math test score fouryears after application

0407(0025)

0183(0027)

0364(0011)

Maximum number of observations 1544 1219 5318

Standard errors are in parentheses For applicants to the choice schools the lsquolsquorstrsquorsquo test score is thelsquolsquopreapplication test scorersquorsquo (the test score from the year of application) or an earlier one if the application testscore is missing for students in the Milwaukee public schools the lsquolsquorstrsquorsquo test score is the rst nonmissing(post-1989) test score See the Data Appendix or Section III for a description of the test scores See the DataAppendix for how I construct a lsquolsquoyear of applicationrsquorsquo for the Milwaukee public schools sample This sample onlyincludes one observation per student and is based on all available data (it does not represent my analysissample)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 555

scores are nationally normed suggesting that the students whoapplied to choice were scoring considerably lower than the na-tional average as well3 On the other hand the parental educa-tion for choice applicants at least for those responding to thesurveys was comparable to (or even a little higher than) nonappli-cants from the Milwaukee public schools

As the program limited participation to independent secularprivate schools the participating schools were not representativeof all private schools in Milwaukee where the majority of privateschool enrollments are likely in religious schools Witte Thornand Pritchard 1995 That said until the constitutionality ofwhether religious schools can participate in voucher programs hasbeen decided the experience in Milwaukee will be relevant forother cities considering such reforms In the rst year sevenprivate schools participated by 1995 this number had risen totwelve These schools represented a variety of educational ap-proaches including Montessori Waldorf bilingual and African-American cultural emphases These were not elite private schoolsFor example the voucher was worth approximately $3200 in1994ndash1995 which contrasts with a range of tuition and fees forschools participating in the choice program of $1080ndash$4000 (in1993ndash1994) In fact the voucher program helped to improve thenancial status of several of the participating schools While onecould believe a priori that a program providing vouchers to eliteprivate schools would result in higher achievement gains it is notso clear that a program providing vouchers to local nonsectarianprivate schools (that are willing to participate) would have sucheffects

Finally although in the Milwaukee Parental Choice Programstate aid followed the students from the public schools to theprivate schools the choice program was too small to provideinsight into the potential general equilibrium student achieve-ment benets of large-scale lsquolsquoschool choicersquorsquo4 In the most unre-stricted school choice program all (or a substantial fraction) of thestudents in the public schools would be eligible to attend a privateschool Since state funding would be tied to student enrollmentsthe public schools would have an incentive to improve leading tono differences in the outcomes of students in lsquolsquopublicrsquorsquo and lsquolsquoprivatersquorsquo

3 See the Data Appendix or Section III for a description of the test scores4 In addition choice schools were only allowed to admit up to 49 percent of

their students as part of the choice program (this level was raised to 65 percent in1994) which limited any potential supply response by private schools

QUARTERLY JOURNAL OF ECONOMICS556

schools in the long run5 Analysis of the Milwaukee ParentalChoice Program can however indicate whether parents wouldprefer to send their children to a private school (see Witte Sterrand Thorn 1995) and whether in the short run the academicachievement of those children who are selected for the programand those who attend the private schools would likely increase

The original evaluation of the fourth year of the choiceprogram conducted by Witte Sterr and Thorn 1995 comparesthe test scores of students in the choice schools with those of arandom sample of all Milwaukee public schools students andwith low-income students and concludes that there were nostatistically signicant relative achievement gains among thechoice students (see also Witte 1997) A subsequent analysis byGreene Peterson Du Boeger and Frazier 1996 criticizes theWitte Sterr and Thorn study for using a comparison group thatwas from substantially more advantaged families than studentsin the choice program Although Witte Sterr and Thorn arguethat their comparison group is in fact comparable to choicestudents conditional on observable covariates it is neverthelesspossible that unobserved factors remain which would tend toobscure any relative achievement gains among the choice studentpopulation

As an alternative to the use of a general comparison groupGreene et al 1996 propose the use of the unsuccessful applicantsas a lsquolsquoquasi-experimentalrsquorsquo control group Relative to the unsuccess-ful applicants Greene et al conclude that the choice studentsmade statistically signicant test score gains by their third andfourth years in the program in both reading and math Howeverthe analysis by Greene et al may overstate the effect of theprogram by excluding from the choice group students who weresuccessfully admitted to the choice schools but did not attendthem or attended only for a short period of time In addition theunsuccessful applicants may not provide an ideal control groupsince those who remained in the Milwaukee public schools appearto have been a nonrandom subset of all unsuccessful applicants

In this paper I use both the unsuccessful applicants and therandom sample of students from the Milwaukee public schools ascomparison groups In addition I return to the lsquolsquotruersquorsquo source ofexogenous variation in the Milwaukee Choice programmdashthat of

5 See Epple and Romano 1996 and Nechyba 1996 for theoretical models ofthe political economy and achievement effects of school vouchers and Hoxby 1996for an (indirect) empirical study

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 557

selection into the pool of students eligible to attend a choiceschool I argue that a complete evaluation of the programrsquos effectson student achievement should consider not only whether privateschools are better than public schools (as do Greene et al 19961997 Witte Sterr and Thorn 1995 and Witte 1997) but alsowhether students who were selected to attend a choice schoolenrolled and remained there I do so by estimating the effect ofbeing selected to attend a choice school on student achievementBecause the unsuccessful applicants are a potentially problematiccontrol group I also compare the test score gains of studentsselected to the choice program with the gains of a random sampleof students in the Milwaukee public schools To control fortime-invariant differences between students selected for theprogram and the comparison groups I implement an individualxed-effects strategy Finally I estimated structural equations ofthe relative effectiveness of the choice schools and the publicschools in Milwaukee

I nd that students selected for the choice program scoredapproximately 15ndash23 extra percentile points per year in mathcompared with unsuccessful applicants and the sample of otherstudents in the Milwaukee public schools The achievement gainsof those actually enrolled in the choice schools were quite similarGiven a (within-sample) standard deviation of about nineteenpercentile points on the math test this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions I do notestimate statistically signicant differences between sectors inreading scores

Some have argued that the key difference between theexisting analyses by Greene et al and Witte et al is thecontrolcomparison group However I nd that the two compari-son groups yield similar estimates when individual xed-effectsare included although the results using the unsuccessful appli-cants are not as robust to variation in sample My results on theeffects on reading scores differ from those reported by Greene etal because their reading results are not robust to the inclusion ofindividual xed-effects and to alternative specications My re-sults for math differ from those reported by Witte partly becauseof our specications and partly because my xed-effects specica-tion takes advantage of a larger sample

Finally these data are not ideal and the problems threatenthe validity of any evaluation of the Milwaukee Parental Choice

QUARTERLY JOURNAL OF ECONOMICS558

Program I have found my results to be robust to substantialmissing and imputed data and (potentially) nonrandom selectionand attrition from the sample Nevertheless because econometrictechniques cannot substitute for better data these data decien-cies should be kept in mind when interpreting the results

II EMPIRICAL STRATEGIES

A Using the Unsuccessful Applicants as a Control Group

In an lsquolsquoidealrsquorsquo experiment the random assignment processcompletely determines the status of individuals in the treatmentand control groups In many experimental settings involvinghuman subjects however there is slippage between the randomassignment status of experimental subjects and whether or notthey actually receive the treatment For example in randomizedtrials of medicines some individuals in the treatment group mayfail to take the prescribed treatment (see for example CoronaryDrug Project 1980 and Efron and Feldman 1991) In somesocial experiments the slippage can occur in both directions Forexample in a job training experiment some people assigned tothe treatment group may fail to show up whereas others in thecontrol group may receive training from other providers Haus-man and Wise 1985 Heckman and Smith 1994 In the speciccontext of the Milwaukee Choice program the slippage betweenrandom assignment statusmdashwhether the individual was chosento attend a choice schoolmdashand lsquolsquotreatment statusrsquorsquomdashwhether anindividual actually attended a choice schoolmdashis signicant be-cause the treatment lasted over several years In fact only about50 percent of students who were selected to attend choice schoolsin the rst year of the program were still attending the school twoyears later In order to understand the nature of alternativeestimates of the relative effect of the choice program in thissetting it is necessary to develop a model that takes account of thelink between the random selection process and the decision toattend a choice school

Consider the following linear probability model that modelschild irsquos actual attendance at a choice school in year t Pit (Pit 5 1for students who attend a choice school Pit 5 0 for those who donot) as a function of irsquos selection status in the previous year t 2 1Sit 2 1 (Sit 2 1 5 1 for students who were selected in year t 2 1Sit 2 1 5 0 for those who were either never selected or were selected

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 559

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 3: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

TABLE IMEAN CHARACTERISTICS OF APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS IN

THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee

public schoolssampleSelected

Not-selected

African-American 0721 0780 0550(0011) (0012) (0007)

Hispanic 0217 0154 0097(0011) (0011) (0004)

lsquolsquoFirstrsquorsquo math score (national percentileranking)

34560(1196)

35073(1552)

43353(0494)

Proportion missing lsquolsquorstrsquorsquo math testscore

0704(0012)

0779(0012)

0382(0007)

lsquolsquoFirstrsquorsquo reading score (national percen-tile ranking)

33869(1050)

34341(1369)

40227(0462)

Proportion missing lsquolsquorstrsquorsquo readingscore

0699(0012)

0781(0012)

0382(0007)

Family income ( 4 1000) (1994 dollars) $12123(0297)

$12715(0509)

$23897(0534)

Proportion missing family income 0491(0013)

0760(0012)

0727(0006)

Motherrsquos education 12559 12399 12108(0067) (0109) (0055)

Proportion missing motherrsquos education 0555(0013)

0763(0012)

0724(0006)

Fatherrsquos education 12048 11521 12250(0092) (0152) (0068)

Proportion missing fatherrsquos education 0697(0012)

0846(0010)

0796(0005)

Grade of application 2997 3875 3481(0079) (0119) (0058)

Proportion with math test score twoyears after application

0536(0015)

0284(0019)

0535(0009)

Proportion with math test score threeyears after application

0467(0017)

0236(0024)

0405(0009)

Proportion with math test score fouryears after application

0407(0025)

0183(0027)

0364(0011)

Maximum number of observations 1544 1219 5318

Standard errors are in parentheses For applicants to the choice schools the lsquolsquorstrsquorsquo test score is thelsquolsquopreapplication test scorersquorsquo (the test score from the year of application) or an earlier one if the application testscore is missing for students in the Milwaukee public schools the lsquolsquorstrsquorsquo test score is the rst nonmissing(post-1989) test score See the Data Appendix or Section III for a description of the test scores See the DataAppendix for how I construct a lsquolsquoyear of applicationrsquorsquo for the Milwaukee public schools sample This sample onlyincludes one observation per student and is based on all available data (it does not represent my analysissample)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 555

scores are nationally normed suggesting that the students whoapplied to choice were scoring considerably lower than the na-tional average as well3 On the other hand the parental educa-tion for choice applicants at least for those responding to thesurveys was comparable to (or even a little higher than) nonappli-cants from the Milwaukee public schools

As the program limited participation to independent secularprivate schools the participating schools were not representativeof all private schools in Milwaukee where the majority of privateschool enrollments are likely in religious schools Witte Thornand Pritchard 1995 That said until the constitutionality ofwhether religious schools can participate in voucher programs hasbeen decided the experience in Milwaukee will be relevant forother cities considering such reforms In the rst year sevenprivate schools participated by 1995 this number had risen totwelve These schools represented a variety of educational ap-proaches including Montessori Waldorf bilingual and African-American cultural emphases These were not elite private schoolsFor example the voucher was worth approximately $3200 in1994ndash1995 which contrasts with a range of tuition and fees forschools participating in the choice program of $1080ndash$4000 (in1993ndash1994) In fact the voucher program helped to improve thenancial status of several of the participating schools While onecould believe a priori that a program providing vouchers to eliteprivate schools would result in higher achievement gains it is notso clear that a program providing vouchers to local nonsectarianprivate schools (that are willing to participate) would have sucheffects

Finally although in the Milwaukee Parental Choice Programstate aid followed the students from the public schools to theprivate schools the choice program was too small to provideinsight into the potential general equilibrium student achieve-ment benets of large-scale lsquolsquoschool choicersquorsquo4 In the most unre-stricted school choice program all (or a substantial fraction) of thestudents in the public schools would be eligible to attend a privateschool Since state funding would be tied to student enrollmentsthe public schools would have an incentive to improve leading tono differences in the outcomes of students in lsquolsquopublicrsquorsquo and lsquolsquoprivatersquorsquo

3 See the Data Appendix or Section III for a description of the test scores4 In addition choice schools were only allowed to admit up to 49 percent of

their students as part of the choice program (this level was raised to 65 percent in1994) which limited any potential supply response by private schools

QUARTERLY JOURNAL OF ECONOMICS556

schools in the long run5 Analysis of the Milwaukee ParentalChoice Program can however indicate whether parents wouldprefer to send their children to a private school (see Witte Sterrand Thorn 1995) and whether in the short run the academicachievement of those children who are selected for the programand those who attend the private schools would likely increase

The original evaluation of the fourth year of the choiceprogram conducted by Witte Sterr and Thorn 1995 comparesthe test scores of students in the choice schools with those of arandom sample of all Milwaukee public schools students andwith low-income students and concludes that there were nostatistically signicant relative achievement gains among thechoice students (see also Witte 1997) A subsequent analysis byGreene Peterson Du Boeger and Frazier 1996 criticizes theWitte Sterr and Thorn study for using a comparison group thatwas from substantially more advantaged families than studentsin the choice program Although Witte Sterr and Thorn arguethat their comparison group is in fact comparable to choicestudents conditional on observable covariates it is neverthelesspossible that unobserved factors remain which would tend toobscure any relative achievement gains among the choice studentpopulation

As an alternative to the use of a general comparison groupGreene et al 1996 propose the use of the unsuccessful applicantsas a lsquolsquoquasi-experimentalrsquorsquo control group Relative to the unsuccess-ful applicants Greene et al conclude that the choice studentsmade statistically signicant test score gains by their third andfourth years in the program in both reading and math Howeverthe analysis by Greene et al may overstate the effect of theprogram by excluding from the choice group students who weresuccessfully admitted to the choice schools but did not attendthem or attended only for a short period of time In addition theunsuccessful applicants may not provide an ideal control groupsince those who remained in the Milwaukee public schools appearto have been a nonrandom subset of all unsuccessful applicants

In this paper I use both the unsuccessful applicants and therandom sample of students from the Milwaukee public schools ascomparison groups In addition I return to the lsquolsquotruersquorsquo source ofexogenous variation in the Milwaukee Choice programmdashthat of

5 See Epple and Romano 1996 and Nechyba 1996 for theoretical models ofthe political economy and achievement effects of school vouchers and Hoxby 1996for an (indirect) empirical study

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 557

selection into the pool of students eligible to attend a choiceschool I argue that a complete evaluation of the programrsquos effectson student achievement should consider not only whether privateschools are better than public schools (as do Greene et al 19961997 Witte Sterr and Thorn 1995 and Witte 1997) but alsowhether students who were selected to attend a choice schoolenrolled and remained there I do so by estimating the effect ofbeing selected to attend a choice school on student achievementBecause the unsuccessful applicants are a potentially problematiccontrol group I also compare the test score gains of studentsselected to the choice program with the gains of a random sampleof students in the Milwaukee public schools To control fortime-invariant differences between students selected for theprogram and the comparison groups I implement an individualxed-effects strategy Finally I estimated structural equations ofthe relative effectiveness of the choice schools and the publicschools in Milwaukee

I nd that students selected for the choice program scoredapproximately 15ndash23 extra percentile points per year in mathcompared with unsuccessful applicants and the sample of otherstudents in the Milwaukee public schools The achievement gainsof those actually enrolled in the choice schools were quite similarGiven a (within-sample) standard deviation of about nineteenpercentile points on the math test this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions I do notestimate statistically signicant differences between sectors inreading scores

Some have argued that the key difference between theexisting analyses by Greene et al and Witte et al is thecontrolcomparison group However I nd that the two compari-son groups yield similar estimates when individual xed-effectsare included although the results using the unsuccessful appli-cants are not as robust to variation in sample My results on theeffects on reading scores differ from those reported by Greene etal because their reading results are not robust to the inclusion ofindividual xed-effects and to alternative specications My re-sults for math differ from those reported by Witte partly becauseof our specications and partly because my xed-effects specica-tion takes advantage of a larger sample

Finally these data are not ideal and the problems threatenthe validity of any evaluation of the Milwaukee Parental Choice

QUARTERLY JOURNAL OF ECONOMICS558

Program I have found my results to be robust to substantialmissing and imputed data and (potentially) nonrandom selectionand attrition from the sample Nevertheless because econometrictechniques cannot substitute for better data these data decien-cies should be kept in mind when interpreting the results

II EMPIRICAL STRATEGIES

A Using the Unsuccessful Applicants as a Control Group

In an lsquolsquoidealrsquorsquo experiment the random assignment processcompletely determines the status of individuals in the treatmentand control groups In many experimental settings involvinghuman subjects however there is slippage between the randomassignment status of experimental subjects and whether or notthey actually receive the treatment For example in randomizedtrials of medicines some individuals in the treatment group mayfail to take the prescribed treatment (see for example CoronaryDrug Project 1980 and Efron and Feldman 1991) In somesocial experiments the slippage can occur in both directions Forexample in a job training experiment some people assigned tothe treatment group may fail to show up whereas others in thecontrol group may receive training from other providers Haus-man and Wise 1985 Heckman and Smith 1994 In the speciccontext of the Milwaukee Choice program the slippage betweenrandom assignment statusmdashwhether the individual was chosento attend a choice schoolmdashand lsquolsquotreatment statusrsquorsquomdashwhether anindividual actually attended a choice schoolmdashis signicant be-cause the treatment lasted over several years In fact only about50 percent of students who were selected to attend choice schoolsin the rst year of the program were still attending the school twoyears later In order to understand the nature of alternativeestimates of the relative effect of the choice program in thissetting it is necessary to develop a model that takes account of thelink between the random selection process and the decision toattend a choice school

Consider the following linear probability model that modelschild irsquos actual attendance at a choice school in year t Pit (Pit 5 1for students who attend a choice school Pit 5 0 for those who donot) as a function of irsquos selection status in the previous year t 2 1Sit 2 1 (Sit 2 1 5 1 for students who were selected in year t 2 1Sit 2 1 5 0 for those who were either never selected or were selected

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 559

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 4: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

scores are nationally normed suggesting that the students whoapplied to choice were scoring considerably lower than the na-tional average as well3 On the other hand the parental educa-tion for choice applicants at least for those responding to thesurveys was comparable to (or even a little higher than) nonappli-cants from the Milwaukee public schools

As the program limited participation to independent secularprivate schools the participating schools were not representativeof all private schools in Milwaukee where the majority of privateschool enrollments are likely in religious schools Witte Thornand Pritchard 1995 That said until the constitutionality ofwhether religious schools can participate in voucher programs hasbeen decided the experience in Milwaukee will be relevant forother cities considering such reforms In the rst year sevenprivate schools participated by 1995 this number had risen totwelve These schools represented a variety of educational ap-proaches including Montessori Waldorf bilingual and African-American cultural emphases These were not elite private schoolsFor example the voucher was worth approximately $3200 in1994ndash1995 which contrasts with a range of tuition and fees forschools participating in the choice program of $1080ndash$4000 (in1993ndash1994) In fact the voucher program helped to improve thenancial status of several of the participating schools While onecould believe a priori that a program providing vouchers to eliteprivate schools would result in higher achievement gains it is notso clear that a program providing vouchers to local nonsectarianprivate schools (that are willing to participate) would have sucheffects

Finally although in the Milwaukee Parental Choice Programstate aid followed the students from the public schools to theprivate schools the choice program was too small to provideinsight into the potential general equilibrium student achieve-ment benets of large-scale lsquolsquoschool choicersquorsquo4 In the most unre-stricted school choice program all (or a substantial fraction) of thestudents in the public schools would be eligible to attend a privateschool Since state funding would be tied to student enrollmentsthe public schools would have an incentive to improve leading tono differences in the outcomes of students in lsquolsquopublicrsquorsquo and lsquolsquoprivatersquorsquo

3 See the Data Appendix or Section III for a description of the test scores4 In addition choice schools were only allowed to admit up to 49 percent of

their students as part of the choice program (this level was raised to 65 percent in1994) which limited any potential supply response by private schools

QUARTERLY JOURNAL OF ECONOMICS556

schools in the long run5 Analysis of the Milwaukee ParentalChoice Program can however indicate whether parents wouldprefer to send their children to a private school (see Witte Sterrand Thorn 1995) and whether in the short run the academicachievement of those children who are selected for the programand those who attend the private schools would likely increase

The original evaluation of the fourth year of the choiceprogram conducted by Witte Sterr and Thorn 1995 comparesthe test scores of students in the choice schools with those of arandom sample of all Milwaukee public schools students andwith low-income students and concludes that there were nostatistically signicant relative achievement gains among thechoice students (see also Witte 1997) A subsequent analysis byGreene Peterson Du Boeger and Frazier 1996 criticizes theWitte Sterr and Thorn study for using a comparison group thatwas from substantially more advantaged families than studentsin the choice program Although Witte Sterr and Thorn arguethat their comparison group is in fact comparable to choicestudents conditional on observable covariates it is neverthelesspossible that unobserved factors remain which would tend toobscure any relative achievement gains among the choice studentpopulation

As an alternative to the use of a general comparison groupGreene et al 1996 propose the use of the unsuccessful applicantsas a lsquolsquoquasi-experimentalrsquorsquo control group Relative to the unsuccess-ful applicants Greene et al conclude that the choice studentsmade statistically signicant test score gains by their third andfourth years in the program in both reading and math Howeverthe analysis by Greene et al may overstate the effect of theprogram by excluding from the choice group students who weresuccessfully admitted to the choice schools but did not attendthem or attended only for a short period of time In addition theunsuccessful applicants may not provide an ideal control groupsince those who remained in the Milwaukee public schools appearto have been a nonrandom subset of all unsuccessful applicants

In this paper I use both the unsuccessful applicants and therandom sample of students from the Milwaukee public schools ascomparison groups In addition I return to the lsquolsquotruersquorsquo source ofexogenous variation in the Milwaukee Choice programmdashthat of

5 See Epple and Romano 1996 and Nechyba 1996 for theoretical models ofthe political economy and achievement effects of school vouchers and Hoxby 1996for an (indirect) empirical study

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 557

selection into the pool of students eligible to attend a choiceschool I argue that a complete evaluation of the programrsquos effectson student achievement should consider not only whether privateschools are better than public schools (as do Greene et al 19961997 Witte Sterr and Thorn 1995 and Witte 1997) but alsowhether students who were selected to attend a choice schoolenrolled and remained there I do so by estimating the effect ofbeing selected to attend a choice school on student achievementBecause the unsuccessful applicants are a potentially problematiccontrol group I also compare the test score gains of studentsselected to the choice program with the gains of a random sampleof students in the Milwaukee public schools To control fortime-invariant differences between students selected for theprogram and the comparison groups I implement an individualxed-effects strategy Finally I estimated structural equations ofthe relative effectiveness of the choice schools and the publicschools in Milwaukee

I nd that students selected for the choice program scoredapproximately 15ndash23 extra percentile points per year in mathcompared with unsuccessful applicants and the sample of otherstudents in the Milwaukee public schools The achievement gainsof those actually enrolled in the choice schools were quite similarGiven a (within-sample) standard deviation of about nineteenpercentile points on the math test this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions I do notestimate statistically signicant differences between sectors inreading scores

Some have argued that the key difference between theexisting analyses by Greene et al and Witte et al is thecontrolcomparison group However I nd that the two compari-son groups yield similar estimates when individual xed-effectsare included although the results using the unsuccessful appli-cants are not as robust to variation in sample My results on theeffects on reading scores differ from those reported by Greene etal because their reading results are not robust to the inclusion ofindividual xed-effects and to alternative specications My re-sults for math differ from those reported by Witte partly becauseof our specications and partly because my xed-effects specica-tion takes advantage of a larger sample

Finally these data are not ideal and the problems threatenthe validity of any evaluation of the Milwaukee Parental Choice

QUARTERLY JOURNAL OF ECONOMICS558

Program I have found my results to be robust to substantialmissing and imputed data and (potentially) nonrandom selectionand attrition from the sample Nevertheless because econometrictechniques cannot substitute for better data these data decien-cies should be kept in mind when interpreting the results

II EMPIRICAL STRATEGIES

A Using the Unsuccessful Applicants as a Control Group

In an lsquolsquoidealrsquorsquo experiment the random assignment processcompletely determines the status of individuals in the treatmentand control groups In many experimental settings involvinghuman subjects however there is slippage between the randomassignment status of experimental subjects and whether or notthey actually receive the treatment For example in randomizedtrials of medicines some individuals in the treatment group mayfail to take the prescribed treatment (see for example CoronaryDrug Project 1980 and Efron and Feldman 1991) In somesocial experiments the slippage can occur in both directions Forexample in a job training experiment some people assigned tothe treatment group may fail to show up whereas others in thecontrol group may receive training from other providers Haus-man and Wise 1985 Heckman and Smith 1994 In the speciccontext of the Milwaukee Choice program the slippage betweenrandom assignment statusmdashwhether the individual was chosento attend a choice schoolmdashand lsquolsquotreatment statusrsquorsquomdashwhether anindividual actually attended a choice schoolmdashis signicant be-cause the treatment lasted over several years In fact only about50 percent of students who were selected to attend choice schoolsin the rst year of the program were still attending the school twoyears later In order to understand the nature of alternativeestimates of the relative effect of the choice program in thissetting it is necessary to develop a model that takes account of thelink between the random selection process and the decision toattend a choice school

Consider the following linear probability model that modelschild irsquos actual attendance at a choice school in year t Pit (Pit 5 1for students who attend a choice school Pit 5 0 for those who donot) as a function of irsquos selection status in the previous year t 2 1Sit 2 1 (Sit 2 1 5 1 for students who were selected in year t 2 1Sit 2 1 5 0 for those who were either never selected or were selected

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 559

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 5: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

schools in the long run5 Analysis of the Milwaukee ParentalChoice Program can however indicate whether parents wouldprefer to send their children to a private school (see Witte Sterrand Thorn 1995) and whether in the short run the academicachievement of those children who are selected for the programand those who attend the private schools would likely increase

The original evaluation of the fourth year of the choiceprogram conducted by Witte Sterr and Thorn 1995 comparesthe test scores of students in the choice schools with those of arandom sample of all Milwaukee public schools students andwith low-income students and concludes that there were nostatistically signicant relative achievement gains among thechoice students (see also Witte 1997) A subsequent analysis byGreene Peterson Du Boeger and Frazier 1996 criticizes theWitte Sterr and Thorn study for using a comparison group thatwas from substantially more advantaged families than studentsin the choice program Although Witte Sterr and Thorn arguethat their comparison group is in fact comparable to choicestudents conditional on observable covariates it is neverthelesspossible that unobserved factors remain which would tend toobscure any relative achievement gains among the choice studentpopulation

As an alternative to the use of a general comparison groupGreene et al 1996 propose the use of the unsuccessful applicantsas a lsquolsquoquasi-experimentalrsquorsquo control group Relative to the unsuccess-ful applicants Greene et al conclude that the choice studentsmade statistically signicant test score gains by their third andfourth years in the program in both reading and math Howeverthe analysis by Greene et al may overstate the effect of theprogram by excluding from the choice group students who weresuccessfully admitted to the choice schools but did not attendthem or attended only for a short period of time In addition theunsuccessful applicants may not provide an ideal control groupsince those who remained in the Milwaukee public schools appearto have been a nonrandom subset of all unsuccessful applicants

In this paper I use both the unsuccessful applicants and therandom sample of students from the Milwaukee public schools ascomparison groups In addition I return to the lsquolsquotruersquorsquo source ofexogenous variation in the Milwaukee Choice programmdashthat of

5 See Epple and Romano 1996 and Nechyba 1996 for theoretical models ofthe political economy and achievement effects of school vouchers and Hoxby 1996for an (indirect) empirical study

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 557

selection into the pool of students eligible to attend a choiceschool I argue that a complete evaluation of the programrsquos effectson student achievement should consider not only whether privateschools are better than public schools (as do Greene et al 19961997 Witte Sterr and Thorn 1995 and Witte 1997) but alsowhether students who were selected to attend a choice schoolenrolled and remained there I do so by estimating the effect ofbeing selected to attend a choice school on student achievementBecause the unsuccessful applicants are a potentially problematiccontrol group I also compare the test score gains of studentsselected to the choice program with the gains of a random sampleof students in the Milwaukee public schools To control fortime-invariant differences between students selected for theprogram and the comparison groups I implement an individualxed-effects strategy Finally I estimated structural equations ofthe relative effectiveness of the choice schools and the publicschools in Milwaukee

I nd that students selected for the choice program scoredapproximately 15ndash23 extra percentile points per year in mathcompared with unsuccessful applicants and the sample of otherstudents in the Milwaukee public schools The achievement gainsof those actually enrolled in the choice schools were quite similarGiven a (within-sample) standard deviation of about nineteenpercentile points on the math test this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions I do notestimate statistically signicant differences between sectors inreading scores

Some have argued that the key difference between theexisting analyses by Greene et al and Witte et al is thecontrolcomparison group However I nd that the two compari-son groups yield similar estimates when individual xed-effectsare included although the results using the unsuccessful appli-cants are not as robust to variation in sample My results on theeffects on reading scores differ from those reported by Greene etal because their reading results are not robust to the inclusion ofindividual xed-effects and to alternative specications My re-sults for math differ from those reported by Witte partly becauseof our specications and partly because my xed-effects specica-tion takes advantage of a larger sample

Finally these data are not ideal and the problems threatenthe validity of any evaluation of the Milwaukee Parental Choice

QUARTERLY JOURNAL OF ECONOMICS558

Program I have found my results to be robust to substantialmissing and imputed data and (potentially) nonrandom selectionand attrition from the sample Nevertheless because econometrictechniques cannot substitute for better data these data decien-cies should be kept in mind when interpreting the results

II EMPIRICAL STRATEGIES

A Using the Unsuccessful Applicants as a Control Group

In an lsquolsquoidealrsquorsquo experiment the random assignment processcompletely determines the status of individuals in the treatmentand control groups In many experimental settings involvinghuman subjects however there is slippage between the randomassignment status of experimental subjects and whether or notthey actually receive the treatment For example in randomizedtrials of medicines some individuals in the treatment group mayfail to take the prescribed treatment (see for example CoronaryDrug Project 1980 and Efron and Feldman 1991) In somesocial experiments the slippage can occur in both directions Forexample in a job training experiment some people assigned tothe treatment group may fail to show up whereas others in thecontrol group may receive training from other providers Haus-man and Wise 1985 Heckman and Smith 1994 In the speciccontext of the Milwaukee Choice program the slippage betweenrandom assignment statusmdashwhether the individual was chosento attend a choice schoolmdashand lsquolsquotreatment statusrsquorsquomdashwhether anindividual actually attended a choice schoolmdashis signicant be-cause the treatment lasted over several years In fact only about50 percent of students who were selected to attend choice schoolsin the rst year of the program were still attending the school twoyears later In order to understand the nature of alternativeestimates of the relative effect of the choice program in thissetting it is necessary to develop a model that takes account of thelink between the random selection process and the decision toattend a choice school

Consider the following linear probability model that modelschild irsquos actual attendance at a choice school in year t Pit (Pit 5 1for students who attend a choice school Pit 5 0 for those who donot) as a function of irsquos selection status in the previous year t 2 1Sit 2 1 (Sit 2 1 5 1 for students who were selected in year t 2 1Sit 2 1 5 0 for those who were either never selected or were selected

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 559

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 6: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

selection into the pool of students eligible to attend a choiceschool I argue that a complete evaluation of the programrsquos effectson student achievement should consider not only whether privateschools are better than public schools (as do Greene et al 19961997 Witte Sterr and Thorn 1995 and Witte 1997) but alsowhether students who were selected to attend a choice schoolenrolled and remained there I do so by estimating the effect ofbeing selected to attend a choice school on student achievementBecause the unsuccessful applicants are a potentially problematiccontrol group I also compare the test score gains of studentsselected to the choice program with the gains of a random sampleof students in the Milwaukee public schools To control fortime-invariant differences between students selected for theprogram and the comparison groups I implement an individualxed-effects strategy Finally I estimated structural equations ofthe relative effectiveness of the choice schools and the publicschools in Milwaukee

I nd that students selected for the choice program scoredapproximately 15ndash23 extra percentile points per year in mathcompared with unsuccessful applicants and the sample of otherstudents in the Milwaukee public schools The achievement gainsof those actually enrolled in the choice schools were quite similarGiven a (within-sample) standard deviation of about nineteenpercentile points on the math test this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions I do notestimate statistically signicant differences between sectors inreading scores

Some have argued that the key difference between theexisting analyses by Greene et al and Witte et al is thecontrolcomparison group However I nd that the two compari-son groups yield similar estimates when individual xed-effectsare included although the results using the unsuccessful appli-cants are not as robust to variation in sample My results on theeffects on reading scores differ from those reported by Greene etal because their reading results are not robust to the inclusion ofindividual xed-effects and to alternative specications My re-sults for math differ from those reported by Witte partly becauseof our specications and partly because my xed-effects specica-tion takes advantage of a larger sample

Finally these data are not ideal and the problems threatenthe validity of any evaluation of the Milwaukee Parental Choice

QUARTERLY JOURNAL OF ECONOMICS558

Program I have found my results to be robust to substantialmissing and imputed data and (potentially) nonrandom selectionand attrition from the sample Nevertheless because econometrictechniques cannot substitute for better data these data decien-cies should be kept in mind when interpreting the results

II EMPIRICAL STRATEGIES

A Using the Unsuccessful Applicants as a Control Group

In an lsquolsquoidealrsquorsquo experiment the random assignment processcompletely determines the status of individuals in the treatmentand control groups In many experimental settings involvinghuman subjects however there is slippage between the randomassignment status of experimental subjects and whether or notthey actually receive the treatment For example in randomizedtrials of medicines some individuals in the treatment group mayfail to take the prescribed treatment (see for example CoronaryDrug Project 1980 and Efron and Feldman 1991) In somesocial experiments the slippage can occur in both directions Forexample in a job training experiment some people assigned tothe treatment group may fail to show up whereas others in thecontrol group may receive training from other providers Haus-man and Wise 1985 Heckman and Smith 1994 In the speciccontext of the Milwaukee Choice program the slippage betweenrandom assignment statusmdashwhether the individual was chosento attend a choice schoolmdashand lsquolsquotreatment statusrsquorsquomdashwhether anindividual actually attended a choice schoolmdashis signicant be-cause the treatment lasted over several years In fact only about50 percent of students who were selected to attend choice schoolsin the rst year of the program were still attending the school twoyears later In order to understand the nature of alternativeestimates of the relative effect of the choice program in thissetting it is necessary to develop a model that takes account of thelink between the random selection process and the decision toattend a choice school

Consider the following linear probability model that modelschild irsquos actual attendance at a choice school in year t Pit (Pit 5 1for students who attend a choice school Pit 5 0 for those who donot) as a function of irsquos selection status in the previous year t 2 1Sit 2 1 (Sit 2 1 5 1 for students who were selected in year t 2 1Sit 2 1 5 0 for those who were either never selected or were selected

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 559

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 7: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

Program I have found my results to be robust to substantialmissing and imputed data and (potentially) nonrandom selectionand attrition from the sample Nevertheless because econometrictechniques cannot substitute for better data these data decien-cies should be kept in mind when interpreting the results

II EMPIRICAL STRATEGIES

A Using the Unsuccessful Applicants as a Control Group

In an lsquolsquoidealrsquorsquo experiment the random assignment processcompletely determines the status of individuals in the treatmentand control groups In many experimental settings involvinghuman subjects however there is slippage between the randomassignment status of experimental subjects and whether or notthey actually receive the treatment For example in randomizedtrials of medicines some individuals in the treatment group mayfail to take the prescribed treatment (see for example CoronaryDrug Project 1980 and Efron and Feldman 1991) In somesocial experiments the slippage can occur in both directions Forexample in a job training experiment some people assigned tothe treatment group may fail to show up whereas others in thecontrol group may receive training from other providers Haus-man and Wise 1985 Heckman and Smith 1994 In the speciccontext of the Milwaukee Choice program the slippage betweenrandom assignment statusmdashwhether the individual was chosento attend a choice schoolmdashand lsquolsquotreatment statusrsquorsquomdashwhether anindividual actually attended a choice schoolmdashis signicant be-cause the treatment lasted over several years In fact only about50 percent of students who were selected to attend choice schoolsin the rst year of the program were still attending the school twoyears later In order to understand the nature of alternativeestimates of the relative effect of the choice program in thissetting it is necessary to develop a model that takes account of thelink between the random selection process and the decision toattend a choice school

Consider the following linear probability model that modelschild irsquos actual attendance at a choice school in year t Pit (Pit 5 1for students who attend a choice school Pit 5 0 for those who donot) as a function of irsquos selection status in the previous year t 2 1Sit 2 1 (Sit 2 1 5 1 for students who were selected in year t 2 1Sit 2 1 5 0 for those who were either never selected or were selected

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 559

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 8: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

in year $ t)

(1) Pit 5 a 1 r Sit 2 1 1 Xig 1 Zig8 1 uit

Xi and Zi are vectors of individual characteristics (to be distin-guished later) Note that if individuals were forced to attend theschool to which they were assigned then r 5 1 and a 5 g 5 g8 5 0More generally however studentsrsquo attendance decisions dependon a variety of factors such as their familyrsquos residential locationwhether the school is attended by friends or siblings and so forthIn this case the parameter r may be smaller than 1

In the Milwaukee Parental Choice program schools were notallowed to discriminate in which students they took This wasinterpreted to mean that if the school was oversubscribed for aparticular grade the students would be randomly selected fromamong the applicants6 Therefore the probability of selectionSit 2 1 is only random conditional on the school and grade to whicha student applied This is because applicants to some schools in acertain grade were more likely to be selected than applicants toother schools or grades or both If one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) any estimated effect of the choiceschools could be spurious Formally this implies that the vector ofcontrol characteristics included in equation (1) must includeindicators for the specic lsquolsquoapplication lotteriesrsquorsquo as represented bythe vector Zi

Consider next an outcome equation for the test score of child iin year t Specically assume that the test score of child i in year t(Tit) is determined by

(2) Tit 5 a 1 b Pit 1 Xi g 1 Zi G 1 e it

In equation (2) the parameter b reects the relative effectivenessof choice schools over the public schools attended by students inthe sample conditional on Xi and Zi

Finally it is useful to combine equations (1) and (2) into areduced-form equation

(3) Tit 5 p 0 1 p 1Sit 2 1 1 Xi P 2 1 Zi P 31 vit

Note that p 1 5 r b Thus the reduced-form effect of selection intothe group eligible to attend choice schools is a combination of twoeffects the effect of selection on the relative likelihood of attend-

6 If a choice school was not oversubscribed it was required to take all whoapplied with only a few exclusions

QUARTERLY JOURNAL OF ECONOMICS560

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 9: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

ing a choice school ( r )mdashwhat might be called a lsquolsquotake-up effectrsquorsquoand the lsquolsquotruersquorsquo effect of choice schools on student achievement (b )In the treatment literature (eg Rubin 1974 and Efron andFeldman 1991) the parameter p 1 is referred to as the lsquolsquointention-to-treatrsquorsquo effect

There are at least two reasons why we might be interested inp 1 rather than the constituent parameters r and b First it is theonly policy instrument available to policy makers If the state ofWisconsin decides to provide educational vouchers to all low-income students not all will take advantage of the program andnot all who enroll will remain For example see Table II Of theapproximately 400 students who applied in 1990 and wereselected to attend a choice school 94 percent ever enrolled in achoice school but only 61 percent remained through the rstspring semester By the second spring approximately 48 percentremained There are similar patterns in later cohorts Clearly rcan be substantially less than 1 In the extreme case in which r 50 even if private schools are much better at educating childrenthan the public schools there will be no achievement gains from

TABLE IINUMBERS OF APPLICANTS SELECTIONS AND ENROLLMENTS

Year of lsquolsquorstrsquorsquo application

1990 1991 1992 1993

Number of applicants 583 558 558 559Number selected 376 452 321 395Number ever enrolled in a choice

school in the fall 354 375 280 327Enrolled in a choice school in the

spring of1991 2311992 181 2701993 130 201 1891994 92 148 156 149

The year of lsquolsquorstrsquorsquo application is the rst year that a student ever applied if she was never selected and therst year that a student applied and was selected if she applied more than once lsquolsquoEnrolled in a choice school inthe fallrsquorsquo means that the students were enrolled as of October of the year enrolled in the spring means theytook the Iowa Tests of Basic Skills in a choice school indicating that they were still enrolled in the choice schoolin the spring Note that I can only observe whether a student was still enrolled in the choice program in thespring if he took an achievement test If a student was absent the day of the test this measure has thepotential to inate the attrition from the program To mitigate this problem for this table if a student isobserved to be enrolled the following year I also assume he is enrolled in the current year (thus for example ifa student is observed to be enrolled in the spring of 1992 and he applied in 1990 I assume he was enrolled inthe spring of 1991 as well) This table is constructed from all available data and does not represent my analysissample

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 561

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 10: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

the program Thus the reduced-form estimates reect the overallpotential gains from offering the vouchers Second as in manyexperimental settings the randomization only occurred in theintention-to-treat and as such the reduced-form estimate is theonly unambiguously unbiased estimate that one can obtain froman ordinary least squares (OLS) regression assuming the initialselection was truly random

I employ two strategies for estimating the reduced-formparameter using only the applicants to the choice program FirstI include dummy variables representing the lsquolsquoapplication lotter-iesrsquorsquo as in equation (3) In this case p 1 is the (conditional) meandifference in test scores between those selected and those notselected for the choice program The identifying assumption isthat if the selected students had remained in the Milwaukeepublic schools they would have had the same mean test score asthe unsuccessful applicants conditional on the application lotter-ies Zi As an alternative estimation strategy I include individualxed-effects The reduced-form equation becomes

(4) Tit 5 p 80 1 p 81Sit2 1 1 v i 1 v8it

where v i is a time-invariant individual xed-effect and v8it is aserially uncorrelated error term By including individual xed-effects the estimator becomes a lsquolsquodifference-in-differences rsquorsquo estima-tor in which the change in test scores for those selected to attend achoice school is compared with the change in test scores forunsuccessful applicants Using this strategy I assume that theselected applicants would have had the same growth in test scoresas the unsuccessful applicants if they had not been selected toattend a choice school Note that the individual xed-effectsubsumes the school and grade to which the student applied theyear in which she applied as well as any other time-invariantbackground characteristics of the student The xed-effects estima-tor also has the advantage of not relying on information about theapplication lotteries which are imputed as explained in SectionIII

Both estimators in equations (3) and (4) will generate unbi-ased estimates of the relative effect of the program ( p 1 and p 81) aslong as the error terms vit and v8it are orthogonal to selection tothe program It is useful to consider the restrictiveness of theseassumptions In equation (3) the lsquolsquoapplicant lotteryrsquorsquo estimator willbe biased upward if there are unobserved differences in thetreatment and control groups Although the initial selection may

QUARTERLY JOURNAL OF ECONOMICS562

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 11: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

have been random (conditional on the application lotteries)unobserved differences may emerge between the two groups overtime This is potentially a problem because there is substantialattrition from the sample that may be nonrandom The primaryreason for the attrition is that test scores for students enrolled inpublic schools outside of Milwaukee and other nonchoice privateschools were not collected And if the more motivated parentsamong the unsuccessful applicants were more likely to enrolltheir child in a private school outside of the choice program thenthe estimate of the intention-to-treat in equation (3) will be biasedupward

In fact Witte 1997 argues that a large fraction of studentswho were not selected in the lotteries chose to attend anotherprivate school (not participating in the choice program) This wasmade easier by a parallel privately funded program (Partners forAdvancing Values in Education (PAVE)) that provided scholar-ships for (primarily) religious schools While I do not haveenrollment data that would allow me to identify which of theapplicants eventually enrolled in another private school studentswho leave the Milwaukee public schools do not have any postappli-cation test scores Table IIIa shows the means and standard errorsof background characteristics of students who have at least onepostapplication test score and those who have no postapplicationtest scores by whether the student was selected or not selected toattend a choice school To ease exposition I will refer to those withpostapplication test scores as lsquolsquostayersrsquorsquo and those without postap-plication test scores as lsquolsquoleaversrsquorsquo In most dimensions there is littledifference between the mean characteristics of stayers and leav-ers among the successful applicants As a result it appears likelythat successful applicants with postapplication test scores (stay-ers) are representative of all successful applicants

The results are different for the unsuccessful applicants Themean family income of unsuccessful stayers is about $1300 lessthan the mean family income of unsuccessful leavers And theparental education of unsuccessful stayers is a little lower thanthat of unsuccessful leavers Neither of these differences isstatistically signicant The means caution that the sample ofunsuccessful applicants with which one can easily estimateeducation production functions may not be a random sample of allunsuccessful applicants although the relatively large standarderrors inhibit denitive inference

This potential bias could also increase over time if an

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 563

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 12: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

increasing (and disproportionate) number of parents of unsuccess-ful applicants move out of Milwaukee or elect to send their child toa private school outside of the choice program If so one wouldestimate the (spurious) pattern that signicant differences in testscores only emerge two or three years after application to thechoice program Table IIIb presents the mean characteristics forstudents by their application status and whether they have a testscore four years after application to the program I have alsoincluded means for the Milwaukee public schools sample forcomparison The results in Table IIIb suggest that based onobservable characteristics (among those with nonmissing values)the successful applicants with and without test scores appearquite similar and the differences between unsuccessful appli-

TABLE IIIaMEAN CHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE ANY

POSTAPPLICATION TEST SCORES FOR 1990 AND 1991 APPLICANTS

TO THE CHOICE PROGRAM

Selected Not-selected

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Havepost-

applicationtest score

Have nopost-

applicationtest scores

Family income ( 4 1000)(1994 dollars)

$12124(0424)

$12754(1120)

$11805(1194)

$13074(1200)

Motherrsquos education(years)

12536(0099)

12317(0245)

11926(0258)

12340(0328)

Fatherrsquos education(years)

11982(0142)

11942(0309)

11036(0382)

11491(0401)

Preapplication math(NCE) test score

37700(1226)

32385(3187)

36919(2076)

36765(3261)

Proportion missing familyincome

0428(0020)

0694(0031)

0628(0038)

0637(0039)

Proportion missing moth-errsquos education

0577(0020)

0741(0030)

0628(0038)

0664(0039)

Proportion missingfatherrsquos education

0680(0019)

0801(0027)

0744(0034)

0805(0033)

Proportion missing pre-application math(NCE) tests score

0641(0019)

0940(0016)

0549(0039)

0886(0026)

Maximum number ofobservations

612 216 164 149

Standard errors are in parentheses

QUARTERLY JOURNAL OF ECONOMICS564

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 13: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

TABLE IIIbCHARACTERISTICS OF STUDENTS BY WHETHER THEY HAVE TEST SCORES FOUR YEARS

AFTER APPLICATION FOR APPLICANTS TO THE CHOICE PROGRAM AND STUDENTS

IN THE MILWAUKEE PUBLIC SCHOOLS

ApplicantsMilwaukee public

schools sampleSelected Not-selected

Has testscore

Does nothave test

scoreHas test

score

Does nothave test

scoreHas test

score

Does nothave test

score

Family income( 4 1000) (1994 dol-lars)

$12841(0885)

$12520(0851)

$ 8105(1517)

$14603(1253)

$20428(1231)

$24419(1141)

Motherrsquos education(years)

12489(0149)

12644(0194)

11368(0487)

12137(0295)

11934(0129)

12127(0104)

Preapplication math(NCE) test score

36937(2175)

38338(2489)

38125(5349)

36461(2724)

41914(0953)

41397(0739)

Math (NCE) test score1 year after applica-tion

42781(1820)

38044(1763)

38294(4088)

36787(3177)

47081(0763)

44075(0567)

Math (NCE) test score2 years after appli-cation

40806(1765)

35000(1728)

34333(3106)

39238(3004)

43432(0876)

42849(0772)

Math (NCE) test score3 years after appli-cation

39723(1425)

36632(2327)

36585(2383)

36556(3175)

40322(0815)

41852(0969)

Proportion missingfamily income

0359(0039)

0592(0033)

0579(0081)

0627(0037)

0707(0017)

0715(0013)

Proportion missingmotherrsquos education

0373(0039)

0578(0033)

0553(0082)

0621(0037)

0699(0017)

0705(0013)

Proportion missingpreapplication testscore

0588(0040)

0695(0031)

0579(0081)

0769(0033)

0434(0019)

0495(0014)

Proportion missingmath (NCE) testsscore 1 year afterapplication

0314(0038)

0493(0033)

0553(0082)

0722(0035)

Proportion missingmath (NCE) testscore 2 years afterapplication

0189(0032)

0498(0033)

0289(0075)

0751(0033)

0332(0018)

0459(0014)

Proportion missingmath (NCE) testscore 3 years afterapplication

0157(0029)

0722(0030)

0289(0075)

0846(0028)

0348(0018)

0697(0013)

Maximum number ofobservations

153 223 38 169 701 1226

Standard errors are in parentheses See the Data Appendix for how I construct a lsquolsquoyear of applicationrsquorsquo forthe Milwaukee public schools sample All of the Milwaukee public schools students have a test score one yearafter application by construction

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 565

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 14: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

cants with and without test scores have not changed substantiallyfrom those reported in Table IIIa On the other hand the gap inthe family incomes of the unsuccessful applicants with andwithout test scores widened to almost $6500 by the fourth yearafter application although the difference in maternal educationremained about constant Both of these ndings suggest thatanalysis using unsuccessful applicants as a control group may bebiased toward nding a positive effect of the choice program Onthe other hand there is no systematic pattern to the differences intest scores It must be emphasized that these comparisons arebased on very little data

The potential source of bias in the xed-effects estimator ismore subtle since any xed characteristics (eg more lsquolsquomotivatedrsquorsquoparents) are absorbed by the individual xed-effect However ifunsuccessful applicants with faster test score trajectories weremore likely to attend a private school outside of the choiceprogram than unsuccessful applicants with slower test scoretrajectories then the estimated effect of the program (fromequation (4)) will be biased upward Unfortunately the data arenot rich enough to allow for an individual-specic time trend inaddition to the individual xed-effect As an alternative I use therandom sample of students from the Milwaukee public schools asdescribed in the next section

Finally one problem with an lsquolsquoexperimentalrsquorsquo analysis is thatrandomization must occur at some point in the experiment andbecause in this program randomization was based on an applicantpool both the causal and reduced-form estimates reect the effectof the program or the effect of choice schools relative to publicschools among students interested in attending a private school Itis not necessarily the treatment effect that one would estimate forthe general population Angrist Imbens and Rubin 1996 Heck-man and Smith 1993

B Using the Milwaukee Public Schools Students as aComparison Group

As an alternative to using the unsuccessful applicants as thecomparison group and to judge whether the unsuccessful appli-cants are representative of students from the Milwaukee publicschools I also estimate the reduced-form equation using therandom sample of students from the Milwaukee public schoolsEquation (4) becomes

(5) Tit 5 p 90 1 p 91Sit2 1 1 p 2UAit 2 1 1 v i 1 v9it

QUARTERLY JOURNAL OF ECONOMICS566

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 15: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

where most elements are dened as before UAit2 1 indicateswhether a student was an unsuccessful applicant in year t 2 1and the coefficient p 2 estimates the difference between unsuccess-ful applicants and the students in the Milwaukee public schoolssample

The advantage of this strategy is that students in the samplefrom the Milwaukee public schools were not so obviously inter-ested in leaving the public school system Therefore they mayhave had less of an inclination to attend another private schooloutside of the choice program On the other hand if students whoapplied to the choice program were unrepresentative of allstudents in the Milwaukee public schools particularly if thosewho did not apply to the program did not expect to be well servedthen any estimated effects of the program may be biased upwardAs shown in Table I the students who applied to the choice schoolswere substantially less advantaged than the random sample ofstudents from the Milwaukee public schools One solution is tocontrol for a set of observable characteristics such as familyincome or preapplication test scores Another more generalsolution is to control for the permanent (lsquolsquoxedrsquorsquo) component of testscores v i As noted above allowing for an individual xed-effectwill generate an unbiased estimate of p 91 as long as the error termv9it is orthogonal to selection to the program

One problem that has plagued nonexperimental evaluationsof public-sector training programs however is that individualswho participate in training programs are observed to haveunusually low earnings in the period in which they are selected forthe program Ashenfelter 1975 If this lsquolsquodiprsquorsquo in earnings repre-sents a permanent change for the individual then the xed-effectsestimator will be biased (because this represents a time-varyingindividual component in the error term that is correlated withparticipation) Similarly Witte and Thorn 1996 argue thatstudents who were doing unusually poorly in the Milwaukeepublic schools were more likely to apply to the choice programHowever their assessment is based on a single year of test scoredata (ie a level difference) whereas the bias is induced by achange in the trend over time Although over one-half of thestudents who applied for the Milwaukee choice program appliedto kindergarten through second grade (which mitigates against alsquolsquodiprsquorsquo) and there is very little preapplication test score history Ihave attempted to gauge the extent to which there may have beena preprogram lsquolsquodiprsquorsquo among applicants by regressing the math test

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 567

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 16: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

score on the number of years prior to application and a dummyvariable indicating the year of application to the choice programfor the subset of students with at least one preapplication testscore The point estimates indicate that neither selected nornot-selected applicants had unusually low test scores in the yearof application to the program although the standard errors arelarge

The xed-effects estimator will also be biased if applicantsand students from the Milwaukee public schools had differenttrends in their tests scores Specically if applicants had fastertest score gains than students who remained in the Milwaukeepublic schools then the xed-effect estimate will be biasedupward Therefore I have also compared the preapplicationtrends in test scores for the applicants with the choice program tothe postapplication trends in the Milwaukee public schools sam-ple Such an exercise assumes that the choice program had nospillovers to the public schools Again the magnitudes of the trendcoefficients for the students who applied to the choice programand the Milwaukee public schools sample are quite similar Whilean assessment about unobservables using observable data cannotbe denitive I nd no strong evidence consistent with an upwardbias in the reduced-form parameter when individual xed-effectsare included7

C Estimating Whether the Choice Schools Are Better than theMilwaukee Public Schools

The reduced-form estimate is rather unsatisfying for thoseparents and policy makers who desire an estimate of the benetsthat are likely to accrue to those children who actually enroll andremain in a private school for a specied period of time (the effectof lsquolsquotreatment on the treatedrsquorsquo) That is it does not establishwhether private schools are better than public schools preciselybecause not all who are assigned to attend a private school do so8

Consider the following modication of equation (2)

(6) Tit 5 a 1 b 8Pit 1 Q Dit 1 l UAit 1 v i 1 e it

In equation (6) I have divided the students into four categoriesthose currently enrolled in a choice school (Pit) those selected to

7 All of these results are available on request8 Witte Sterr and Thorn 1995 Witte 1997 and Greene et al 1996 1997

attempt to conduct such an analysis by excluding children who left the choiceprogram and reenrolled in a Milwaukee public school

QUARTERLY JOURNAL OF ECONOMICS568

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 17: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

attend a choice school but who are not currently enrolled in one(Dit) and unsuccessful applicants (UAit) the base group is therandom sample of students from the Milwaukee public schoolsThe categories are mutually exclusive in any one year v i is anindividual xed-effect In this case b 8 represents the test scoregains of students enrolled in a choice school relative to a randomsample of students enrolled in the Milwaukee public schools Q isan estimate of the lsquolsquopartial treatment effectrsquorsquo for those whoattended a choice school for some period of time and l representsthe test score gains of the unsuccessful applicants relative to thestudents in the Milwaukee public schools sample

I use two strategies to establish whether the choice schoolswere lsquolsquobetterrsquorsquo than the Milwaukee public schools The rstfollowing the strategy discussed above includes individual xed-effects and requires all of the assumptions underlying the reduced-form estimates using either the unsuccessful applicants or theMilwaukee public schools students as control (or comparison)groups In addition it requires the assumption that the studentswho leave the choice schools are a random sample of studentsenrolled in the choice schools

As shown in Table IV however it appears that among thestudents who enrolled in a choice school those who left were not arandom sample of the students who remained The rst columnmodels the likelihood that a student left the choice program aftertwo years conditional on having attended a choice school for twoyears the second column models the likelihood that a student leftafter three years conditional on having attended for three yearsApproximately 30 percent of the students left the program in eachyear The most noticeable determinant of whether a student leftthe choice program was his or her current grade The older thestudent the more likely he or she was to leave a choice school9 Inaddition in both models students with higher math test scores inthe current year were less likely to leave the choice program (thep-value of the effect is 007 in the rst column and 014 in thesecond) These results suggest that students who remained in thechoice schools may have been a self-selected group

As an alternative strategy for estimating the lsquolsquocausalrsquorsquo effect ofattending a choice school on student achievement I use the initialselection as an instrumental variable that is correlated with Pit

and uncorrelated with the error term in the education production

9 I have excluded students who reached the terminal grade at the schoolfrom this analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 569

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 18: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

function10 The instrumental variables (IV) estimate of b willprovide a consistent estimate of the lsquolsquocausalrsquorsquo effect of attending achoice school for a period of time on test scores even if those whoremain enrolled in a choice school are self-selected AngristImbens and Rubin 1996

10 Evans and Schwab 1995 Figlio and Stone 1997 Neal 1997 andSander 1996 also implement instrumental variables strategies to estimate thecausal effects of private schools

TABLE IVLINEAR PROBABILITY MODELS OF WHETHER A STUDENT LEAVES THE CHOICE

PROGRAM AFTER TWO OR THREE YEARS (CONDITIONAL ON HAVING ATTENDED FOR

TWO OR THREE YEARS) 1990 AND 1991 COHORTS

Sample

Those with a math(NCE) test score ina choice school two

years after application

Those with a math(NCE) test score ina choice school three

years after application

Family income ( 4 1000) (1994dollars)

2 0001(0004)

0003(0007)

Motherrsquos education (years) 0014(0022)

0009(0031)

Fatherrsquos education (years) 2 0010(0018)

2 0076(0033)

Application grade 0068(0013)

0107(0024)

Preapplication math (NCE) testscore ( 4 10)

0025(0027)

0067(0035)

Math (NCE) test score the yearbefore ( 4 10)

2 0025(0014)

2 0042(0029)

Missing family income 2 0122(0074)

0363(0149)

Missing motherrsquos education 0036(0070)

2 0231(0089)

Missing fatherrsquos education 0167(0067)

0076(0114)

Missing preapplication testscore

0018(0062)

0036(0100)

Mean of dependent variable 0287 0306R2 0133 0298Number of observations 307 108

The dependent variable equals 1 if the student does not enroll in a choice school and does not take a test ina choice school the following year and equals 0 if she enrolls in a choice school or takes a test in a choice schoolthe following year Huber standard errors are in parentheses Both models include an intercept Students whoreach the terminal grade at the school have been excluded

QUARTERLY JOURNAL OF ECONOMICS570

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 19: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

D Modeling Educational Achievement

Another issue to consider is the proper form of the educationproduction function In the most basic equation studentsrsquo out-comes are modeled as a function of schooling inputs such aswhether the student attended a choice school as in equation (2)Instead I estimate the effect of having been in a choice school forsome period of time as it is quite possible that a childrsquos achieve-ment does not improve immediately as she may need some time toadjust to her new environment11 In the unrestricted model Iinclude dummy variables indicating the years since the studentapplied to the choice school and interactions between the yearssince the student applied and whether the student was selected toattend a choice school that is

(7) Tit 5 a 1 ok5 2 3

4

p kYit 2 k 1 ok5 1

4

p kYit2 k 3 Sit 2 k ) 1 Xi g 1 Zi G 1 vit

where Yit 2 k are dummy variables indicating the number of yearspre- or postapplication and Sit 2 k are dummy variables indicatingthat the student had been selected to attend a choice school (itonly equals one in the years after application for those who wereselected) the vector p k reects the unconstrained effect of selec-tion into the program on test scores I also constrain the year-to-year gains to be equal and estimate the linear effect of years in theprogram on student achievement (however note that in mostspecications I allow the trend for the comparison group Yit2 k tobe nonlinear)

(8) Tit 5 a 1 a sSit 1 ok5 2 3

4

p Yit 2 k 1 p (Yit 3 Sit ) 1 Xi g 1 Zi G 1 vit

In this equation a s is a separate intercept for students selected forthe choice program Yit2 k is dened as above and Yit 3 Sit is aninteraction between the number of years since application to theprogram and whether the student was selected (this variable isonly nonzero in years after application) I estimate similarspecications including individual xed-effects The xed-effectsestimator compares the deviations from individual-specic meansfor selected applicants with the deviations from individual-

11 This is the argument advanced by Greene et al 1996 Also according toJohn Witte the private schools in the choice program reported that they were moreconcerned about getting the children interested in learning in the rst couple ofyears than in increasing their test scores

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 571

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 20: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

specic means for the unsuccessful applicants (or the studentsfrom the Milwaukee public schools sample)

III DATA

A Data

The samples I analyze in this paper are drawn from theMilwaukee Parental Choice Program public release data lesWitte and Thorn 1995 These les contain administrative dataon information such as test scores race sex grade level andwhether the student applied and was accepted to the choiceprogram as well as some descriptive information about theMilwaukee public schools the students were attending or at-tended either pre- or postenrolling in a choice school In additionthere is family background information based on respondents to asurvey administered each fall and spring to all rst-time choiceapplicants from the previous year and to families from theMilwaukee public schools sample in 1991 In total there are dataon approximately 2300 applicants to the choice program and on asample of approximately 5300 students from the Milwaukeepublic schools

B Imputing the Application Lotteries

As discussed above the probability of selection to a choiceschool is only random conditional on knowing to which school andgrade a student applied Therefore if one does not control for thelsquolsquoapplication lotteriesrsquorsquo (indicators for the school and grade towhich an individual applied) the estimated effect of the programbased on equation (3) could be spurious And yet the MilwaukeeParental Choice Program public use data do not indicate thechoice school to which a student applied To circumvent thisproblem Greene et al 1996 note that one can infer the lottery inwhich an individual participated by knowing an individualrsquos racegrade to which she applied and year of application They do so byhighlighting that over 80 percent of the choice students wereenrolled in one of three schools Almost all students who applied toone of these schools were Hispanic and almost all students whoapplied to the two others were African-American Thus forAfrican-American and Hispanic students selection can be as-sumed random conditional on a set of dummy variables thatrepresent interactions between a childrsquos race the grade to which

QUARTERLY JOURNAL OF ECONOMICS572

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 21: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

he applied and the year of application12 I have attempted toassess whether the selection was (conditionally) random usingthis imputation strategy by regressing individual characteristicson whether the student was selected and (approximately 72)dummy variables indicating the studentrsquos applicant pool Selec-tion was insignicantly related (at the 5 percent level) to thechildrsquos sex preapplication test score family income and maternaleducation On the other hand selection was statistically signi-cantly related to parental education and whether the data weremissing for the preapplication test score and family income13

The fact that selection does not appear perfectly random isperhaps not surprising given the notable places where slippagebetween the actual application lotteries and these constructedlotteries could theoretically occur First the grade at the schoolmay not have been oversubscribed Second race may not com-pletely determine to which school an individual applied (note inparticular that African-Americans are assumed to have applied toone of two schools) Third students could apply to more than oneschool although I cannot adjust for that with these data Fourththe imputation of the grade at application assumes that studentsprogressed at grade level Finally if a sibling of a student wasalready enrolled in the choice program then the student wasadmitted without having to go through the randomization Be-cause the approximation may not perfectly match the actualapplication lotteries I control for whether the student is femaleand the family income of the student (these constitute theelements in the vector Xi in equations (1)ndash(3))

C The Test Scores

The test scores are the normal curve equivalent (NCE)reading and math scores from the Iowa Tests of Basic Skills14

12 There are 72 application lotteries (2 races 3 9 grades 3 4 years) when allfour application cohorts are used The median number of students in each lottery isabout 28

13 These results are available from the author on request14 Greene et al 1997 adjust the test scores of students who are not

at-grade-level to their lsquolsquoage-appropriatersquorsquo grade They do so to account for the factthat students who are held back have taken the same test twice and are older thantheir classmates and that students who are double-promoted are younger thantheir classmates I have elected not to make such an adjustment because it is notclear that students who are held back and therefore are taking a test for the secondtime should be penalized and that students who are double-promoted should begiven a lsquolsquopoint-handicaprsquorsquo I have estimated all of the equations controlling forwhether a student is at-grade-level and restricting the analysis to those at-grade-level with very similar results (This occurs because the (unconditional) promotionrates are similar between the choice schools and the Milwaukee public schools)

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 573

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 22: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

These tests were administered to the choice students everyspring They were also administered to Chapter I students everyyear Other students in the Milwaukee public schools were testedless frequently (primarily in grades 2 5 and 7) Beginning in1993 schools were no longer required to administer the entirebattery of math subtests in order to receive Chapter I fundsrather they were only required to administer the problem-solvingcomponent15 As a result I do not observe a lsquolsquototal math scorersquorsquo for asubstantial fraction of students in the Milwaukee public schools16

At the same time the public release version of the data do notinclude the problem-solving component of the math tests for thestudents in the choice program Fortunately a subset of studentsin the Milwaukee public schools were administered the entirebattery of math tests Therefore I impute the total math score byregressing the total score on the problem-solving component usingthe random sample of students in the Milwaukee public schoolswho did not apply to choice and who took the entire test (see theData Appendix for the equations) I use the predicted total scorefor those with only the problem-solving component and include adummy variable indicating whether the score was imputed I haveexplored the likely effect of this imputation by comparing it withalternative ways of handling the missing data and conclude thatthe results are robust to the imputation these results areavailable on request

D Analysis Samples

In the initial analysis I use data on students who applied toattend one of the choice schools I outline in detail how Iconstructed the sample in the Data Appendix This analysissample consists of African-American and Hispanic students whoapplied to the choice program between 1990 and 1993 for gradesKndash817

15 And in 1995 the Milwaukee public schools began administering state-mandated tests rather than the Iowa Tests of Basic Skills Witte Sterr and Thorn1995 as such I cannot evaluate the program using the 1995 test scores

16 In 1993 approximately 40 percent of the unsuccessful applicants aremissing the total math score and the percentage rises to 69 percent in 1994Similarly in the Milwaukee public schools sample 34 percent of the students haveimputed math test scores in 1993 and 67 percent have imputed math test scores in1994

17 As the Iowa Tests of Basic Skills does not appear to have been adminis-tered to students in grades 9ndash12 my analysis excludes the private high schoolsthat participated in the program (so do Greene et al 1996 1997 Witte et al1995 and Witte 1997) The high schools were predominantly lsquolsquoalternativersquorsquo highschools designed for at-risk students

QUARTERLY JOURNAL OF ECONOMICS574

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 23: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

In order to incorporate the students in the Milwaukee publicschools sample into the analysis I articially create a lsquolsquoyear ofapplicationrsquorsquo from which to measure annual changes in test scoresI describe my categorization method in the Data Appendix I alsorestrict the sample to African-American and Hispanic students(for consistency with the analysis using the application lotteries)although the results are quite similar when I include all studentsDescriptive statistics for both samples are in Appendix 1

IV RESULTS

A Evaluating the Choice Program Reduced-Form Estimates

The basic reduced-form results using only the applicants tothe choice program are presented in Tables Va and Vb In theodd-numbered columns I present OLS estimates with Huberstandard errors that allow for individual correlations and that arerobust to heteroskedasticity I also include dummy variables forthe lsquolsquoapplication lotteriesrsquorsquo as well as a dummy variable forwhether the student is female family income and a dummyvariable indicating whether the family income is missing Ipresent models that include individual xed-effects in the even-numbered columns In Table Va I also include a dummy variableindicating whether the math test score was imputed Finally inthe top panel I constrain the main effect of years before or afterapplication to be linear and the yearly effect of being selected tothe choice program to be a linear function of years since applica-tion in the middle panel I only constrain the years since applica-tion interacted with being selected to the choice program to belinear and in the bottom panel I allow all trends to be nonlinear Ionly present the coefficients on the interaction terms in the middleand bottom panels

In Table Va the estimates in columns (1) and (2) indicate thatunsuccessful applicants lost about 09 (approximately) percentilepoints each year18 In several columns the intercept for thoseselected to the choice program is slightly lower than that for thecontrol group although the differences are not statistically differ-

18 I do not control for the grade level of the test Because most studentsadvanced one grade each year the variables lsquolsquoyears since applicationrsquorsquo and lsquolsquogradelevel of the testrsquorsquo are highly correlated Importantly however while the inclusion orexclusion of the grade level affects the trend in years since application it has littleeffect on the interaction between years since application and whether a studentwas selected to attend a choice school

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 575

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 24: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

TABLE VaTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON MATH SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

2 0679(1081)

2 1440(0979)

2 2088(1407)

2 2879(1307)

2 0037(1128)

2 1404(1055)

Number of yearsbefore or after appli-cation

2 0931(0394)

2 0944(0358)

2 0558(0500)

2 0678(0411)

2 0750(0423)

2 0753(0378)

Selected 3 number ofyears after applica-tion

1381(0528)

1552(0479)

1565(0777)

2017(0691)

0695(0596)

1227(0556)

p-value of F-test ofconstraints in alltrends

0475 0165 0506 0153 0627 0117

R2 0092 0761 0108 0781 0093 0770

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0441(2171)

0102(1908)

1161(3027)

0856(2752)

1896(2357)

1889(2134)

Selected 3 number ofyears after applica-tion

1510(0955)

1543(0857)

0119(1829)

0790(1628)

2 0077(1176)

0159(1104)

p-value of F-test ofconstraint in trend

0191 0249 0952 0805 0751 0488

R2 0093 0762 0110 0783 0095 0771

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

2001(1567)

2326(1504)

1309(1835)

1731(1738)

1984(1570)

2296(1510)

Selected 3 two yearsafter application

1282(1768)

1815(1733)

1302(2417)

2176(2155)

1337(1775)

1554(1755)

Selected 3 three yearsafter application

2087(2218)

3980(2305)

1692(4199)

3806(3867)

2083(2259)

3024(2359)

Selected 3 four yearsafter application

9977(3173)

8772(3001)

R2 0094 0763 0110 0783 0095 0771Number of observa-

tions3177 3177 2038 2038 3000 3000

The dependent variable is the math (NCE) score Standard errors are in parentheses The OLS columnsreport Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual The OLS regressions include aconstant a dummy variable for female family income and an indicator if income is missing The FE columnsinclude individua l xed-effects Columns (7)ndash(18) also control for unrestricted dummy variables indicating thenumber of years before or after application All regressions include a dummy variable indicating if the testscore was imputed The specications also include 71 applicant pool dummy variables in columns (1) (5) (7)(11) (13) and (17) and 53 dummy variables in columns (3) (9) and (15) The F-tests of the constraints arerelative to the fully unrestricted specications in columns (13)ndash(18)

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS576

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 25: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

TABLE VbTHE EFFECT OF SELECTION TO THE CHOICE PROGRAM ON READING SCORES

OLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES USING ONLY

APPLICANTS TO THE CHOICE PROGRAM

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

(1) (2) (3) (4) (5) (6)

Selected to attendchoice school(selected)

1683(0964)

1172(0906)

2 1423(1311)

2 1673(1231)

1601(1035)

0746(0987)

Number of yearsbefore or after appli-cation

2 0785(0351)

2 0529(0312)

2 0136(0433)

2 0380(0361)

2 0596(0368)

2 0552(0332)

Selected 3 number ofyears after applica-tion

2 0374(0501)

2 0546(0449)

0427(0748)

0750(0663)

2 0565(0571)

2 0256(0527)

p-value of F-test ofconstraints in alltrends

0901 0783 0973 0837 0935 0806

R2 0064 0726 0062 0743 0060 0733

(7) (8) (9) (10) (11) (12)

Selected to attendchoice school(selected)

2 0115(1958)

1747(1768)

2 2284(2831)

2 1420(2552)

0362(2202)

0721(1990)

Selected 3 number ofyears after applica-tion

0605(0883)

2 0589(0779)

0977(1658)

0876(1487)

0241(1107)

0019(1006)

p-value of F-test ofconstraint in trend

0771 0335 0405 0487 0599 0375

R2 0065 0727 0062 0743 0061 0733

(13) (14) (15) (16) (17) (18)

Selected 3 one yearafter application

0926(1434)

1135(1393)

2 0928(1713)

2 0313(1613)

0852(1439)

1025(1407)

Selected 3 two yearsafter application

0277(1645)

0250(1592)

2 1538(2276)

2 0366(1962)

0246(1646)

2 0029(1614)

Selected 3 three yearsafter application

1787(2083)

1880(2092)

2691(3496)

2664(3457)

1696(2077)

1561(2136)

Selected 3 four yearsafter application

3348(2935)

2 2517(2759)

R2 0065 0727 0063 0743 0061 0733Number of observa-

tions3163 3163 2023 2023 2986 2986

The dependent variable is the reading (NCE) score Standard errors are in parentheses The specicationsalso include 70 applicant pool dummy variables in columns (1) (5) (7) (11) (13) and (17) and 52 dummyvariables in columns (3) (9) and (15) See also notes to Table Va

These sample restrictions only apply to the applicants

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 577

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 26: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

ent from zero In contrast the interaction between the number ofyears before or after application and whether the student wasselected to the choice program using the full sample in columns(1) (2) (7) and (8) is positive and statistically signicant (exceptfor column (7)) those selected to the program gained an additional15 (percentile) points over the unsuccessful applicants Howevernote that because of the (often) negative main effect of beingselected to the program the level difference in test scores betweenthose selected for the program and those not selected is notstatistically signicant until two years after application

These effects are disaggregated in the bottom panel of thetable where the coefficients reect the effect of being selected toattend a choice school after one two three or four years In therst column the dummy variables suggest that in the rst threeyears after application selected students scored approximatelytwo percentile points higher than not-selected students althoughthe difference is not statistically signicant The only statisticallysignicant difference emerges in the fourth year when selectedstudents scored ten points higher than the unsuccessful appli-cants The estimates in column (13) are quite similar to thosereported by Greene et al 1996 Recall that the main difference inour analyses is that I include those selected to the program whodid not necessarily enroll in or who left the choice schools The factthat the results are so similar most likely reects the relativelysmall number of former choice students who returned to theMilwaukee public schools and were tested rather than randomattrition from the choice schools19

In columns (13) and (14) the mean difference in test scoresbetween those selected and those not selected for the program ismuch higher in the fourth year than for any other year and thiscoefficient is the only one derived from a single year from a singlecohort (the 1990 applicants) To assess the extent to which theresults might be driven by this cohort or by unusually low testscores in the fourth year among the unsuccessful applicantsbecause of nonrandom attrition I tried using alternative samplesFirst I estimated the equations excluding the 1990 cohortSecond I excluded the 1994 test scores for the 1990 applicantcohort but included their 1991ndash1993 test scores The results arein the remaining columns of Table Va The coefficient estimates in

19 On average approximately 20 percent of those who left the choice schoolshad a nonmissing test score in the Milwaukee Public Schools in any one year

QUARTERLY JOURNAL OF ECONOMICS578

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 27: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

columns (3) and (4) which exclude the 1990 cohort altogether arequite similar to those using the full sample However the resultsthat allow for a nonlinear trend in test scores for the unsuccessfulapplicants in columns (9) and (10) are much less robust to thisexclusion Similarly when I exclude the 1994 test scores for the1990 cohort the coefficient on the yearly increase in test scoresusing the lsquolsquoapplication lotteriesrsquorsquo drops considerably and is nolonger statistically signicant (see columns (5) and (11)) Thecorresponding xed-effects estimate using the most restrictedspecication (in column (6)) remains roughly constant although itdecreases substantially in column (12) The point estimatessuggest a (statistically signicant) yearly increase of 15 (percen-tile) points per year when the full sample is analyzed howeverthey are not robust to excluding the fourth year data for the 1990applicants with the less restrictive specications20

Finally the fact that the point estimates in the lower panelare generally not statistically signicant while those in the upper(two) panels are (often) signicant indicates that much of theefficiency in the upper two panels comes from constraining theeffect of the choice program to be linear I tested whether theseconstraints are rejected by the data the p-values are presented inthe tables Although the trends in the interaction between beingselected for the choice program and years since application in thebottom panel do not appear linear the linear restrictions imposedin the upper two panels are not rejected in any of the specica-tions In addition the p-values of the constraints in the upperpanel suggest that constraining the main effect of years sinceapplication (the test score growth for the unsuccessful applicants)to be linear is also not rejected by the data Nevertheless thesensitivity of the effect of selection to the choice program to thelinear constraint in the base trend (in the upper two panels)illustrates that the mean of the fourth year test scores for theunsuccessful applicants is unusually low

A similar set of results for the reading scores is presented inTable Vb Although the magnitude of the yearly gains for theunsuccessful applicants (ie the coefficients on lsquolsquonumber of yearssince applicationrsquorsquo) are similar to those for the math test scores

20 Witte 1997 argues that results obtained using unsuccessful applicantsas a control group are sensitive to the extraordinarily low math test scores of a fewunsuccessful applicants in the fourth year The fragility of the results to theexclusion of the fourth year test scores for the 1990 cohort supports this conclusionMore generally however I nd that the results are not as sensitive to excluding allstudents who scored 5 or lower on the math test in any year

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 579

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 28: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

the differential gains for students selected for the choice programare often negative and insignicantly different from zero Thepoint estimates and standard errors reported in column (13) areroughly similar to those reported by Greene et al 1996 althoughthey interpret their results differently Specically Greene et alrely on one-tailed t-tests because (they argue) theoretically pri-vate school students should perform better In addition theresults presented in Table Vb indicate that the fourth year effecton reading test scores is not robust to the inclusion of individualxed-effects and that although the linear constraints are notrejected the constrained effect of years since application inter-acted with whether the student was selected is often negative andstatistically insignicant In short the Greene et al results forreading scores are fragile

A natural question is whether the math results are driven bymissing test score data As a rst strategy for assessing thepotential effect of sample attrition on the parameter estimates Irestricted the sample to those without missing test scores (thesample size falls dramatically) I also employed a Heckmantwo-step selection correction Compared with an OLS coefficientestimate of 138 for the effect of being selected to the choiceprogram in Table Va (column (1)) the estimate increases to 231with a standard error of 084 when the sample is restricted to onlythose without missing test scores The estimate using a two-stepHeckman selection correction is 218 with a standard error of07721 While these strategies are only suggestive they indicatethat sample attrition is not driving the results22

As a second strategy I compare the test score progress ofselected students with that of the sample of students from theMilwaukee public schools (as in equation (5)) These specicationsinclude dummy variables indicating whether a student wasselected to attend a choice school and whether a student was notselected to attend a choice school The students in the Milwaukee

21 I excluded from the second stage whether the student was eligible for afree or reduced lunch the distance from the studentrsquos home to the school andinteractions of these variables with whether the student was selected Oneproblem with using a Heckman selection correction in this context is that there isnot one underlying reason for the missing test scores As a result there is no singlelatent variable that may underlie the rst-stage probit These estimates areavailable from the author upon request

22 I have also attempted to determine whether the results are driven byfaster (preapplication) test score trajectories among the applicants by interactingthe years since application with the studentrsquos sex race income preapplication testscore and grade at application The results are unchanged

QUARTERLY JOURNAL OF ECONOMICS580

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 29: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

public schools are the base group In addition I include interac-tions between whether the student was selected or not selectedand the number of years since the student applied I allow the testscore growth of the Milwaukee public schools students to benonlinear Except for the fact that the OLS estimates do notinclude lsquolsquoapplication lotteriesrsquorsquo (Zi) the specications and samplesare the same as those in Tables Va and Vb The results are inTable VI23

In the upper panel in all cases the OLS estimates of theinteraction between whether a student is selected and the numberof years since application are positive and statistically signicantAnd the xed-effects estimates are roughly twice the magnitudeof the OLS estimates (and statistically signicant) The xed-effects estimates suggest that students selected for the choiceprogram earn twondashthree additional percentile points per yearrelative to students in the Milwaukee public schools On the onehand the coefficient on the interaction between whether thestudent was selected and the number of years since application isnot sensitive to the inclusion of the fourth year test scores of the1990 cohort (columns (3)ndash(6)) On the other hand the coefficienton the interaction between whether the student was not selectedand the number of years since application is quite sensitive to theexclusion of the fourth year test scores The coefficient rises whenthis nal year test score is excluded because the fourth year testscores among the unsuccessful applicants dropped considerablyAt the same time the point estimates are not signicantlydifferent from zero in most of the specications suggesting thatthere is little statistical difference between the test score trends ofthe unsuccessful applicants and the Milwaukee public schoolssample The results for reading in the lower panel are also quitesimilar to those obtained using the unsuccessful applicants as thecomparison group

The reduced-form estimates are summarized in Figures I andII These gures graph the coefficient estimates from a regressionof the test score on unrestricted dummies representing thenumber of years before and after the year of application andinteractions between the number of years since application andindicators for whether the student was selected or not selected forthe choice program the math specications also include a dummy

23 The results using a fully unrestricted specication are presented inAppendices 2 and 3

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 581

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 30: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

TABLE VIOLS AND INDIVIDUAL FIXED-EFFECTS (FE) ESTIMATES OF THE EFFECT OF

SELECTION TO THE CHOICE PROGRAM ON MATH AND READING SCORES ESTIMATES

USING THE RANDOM SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A

COMPARISON GROUP

Full sample1991ndash1993

cohorts only

Excludes 1994test scores for1990 cohort

OLS FE OLS FE OLS FE

Dependent variable 5 math (NCE) test score

(1) (2) (3) (4) (5) (6)

Selected to attend choiceschool (selected)

2 1760(1158)

2 3391(1114)

2 1900(1501)

2 4201(1417)

2 1507(1252)

2 3256(1202)

Not selected to attendchoice school

2 2091(2069)

2 3052(1741)

2 5834(2951)

2 4387(2511)

2 4392(2280)

2 4665(1959)

Selected 3 number ofyears since application

1295(0491)

2294(0399)

1695(0751)

2940(0638)

1123(0579)

2163(0508)

Not selected 3 number ofyears since application

0097(0902)

0672(0783)

2937(1774)

1820(1493)

1641(1114)

1905(1020)

p-value of F-test of con-straints in all trends

0399 0321 0681 0599 0868 0245

R2 0016 0761 0016 0766 0016 0764Number of observations 8729 8729 7570 7570 8548 8548

Dependent variable 5 reading (NCE) test score

(7) (8) (9) (10) (11) (12)

Selected to attend choiceschool (selected)

0395(1018)

0750(1027)

2 2038(1323)

2 1555(1319)

0509(1115)

0520(1115)

Not selected to attendchoice school

1317(1838)

2 0644(1584)

0114(2633)

0414(2247)

0599(2054)

0165(1781)

Selected 3 number ofyears since application

2 0184(0430)

2 0249(0368)

0776(0641)

1073(0598)

2 0268(0517)

2 0113(0472)

Not selected 3 number ofyears since application

2 0879(0855)

0247(0706)

0332(1534)

2 0078(1327)

2 0401(1046)

2 0236(0915)

p-value of F-test of con-straints in all trends

0758 0583 0773 0431 0504 0441

R2 0021 0738 0022 0744 0021 0741Number of observations 8751 8751 7592 7592 8569 8569

Standard errors are in parentheses The OLS columns report Huber standard errors that allow forcorrelations lsquolsquowithinrsquorsquo an individual The OLS regressions include a constant a dummy variable for femalefamily income and an indicator if income is missing The FE columns include individua l xed-effects Themath test score regressions also include a dummy variable indicating if the test score was imputed Allregressions control for unrestricted dummy variables indicating the number of years before or afterapplication The F-tests of the constrants are relative to a fully unrestricted specication (see Appendixes 2and 3)

These sample restrictions only apply to the applicants lsquolsquoNot Selectedrsquorsquo indicates that the student applied to the choice program and was not accepted

QUARTERLY JOURNAL OF ECONOMICS582

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 31: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

variable indicating whether the test was imputed All specica-tions also include individual xed-effects The underlying pointestimates and standard errors are reported in Appendix 2

Figure I shows that students selected for the choice programhad nearly linear test score gains in math particularly beginningin the second year The gure also reveals that much of the lsquolsquogainrsquorsquooccurs because both the unsuccessful applicants and the studentsin the Milwaukee public schools samples (both groups of whichare in the Milwaukee public schools) experienced large declinesin their test scores Figure II shows the trends for reading scoresAgain it is clear that there are no differences in the test scoresamong the three groups

These reduced-form estimates are unbiased so long as stu-dents who were selected to the choice program did not havedifferent preapplication test score trajectories than either theunsuccessful applicants or the students in the Milwaukee publicschools sample While descriptive statistics suggest that these arereasonable assumptions there may be residual unobserved differ-ences for which I cannot control Keeping this in mind the resultssuggest that being selected to participate in the choice program

FIGURE IAdjusted Math (NCE) Test Scores by Years Since Application to Choice Program

All CohortsCoefficient estimates from a regression of the math scores on dummy variables

for years since application years since application interacted with whether thestudent was selected to attend a choice school or whether the student was notselected to attend a choice school whether the test score was imputed andindividual xed-effects

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 583

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 32: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

appears to have increased the math achievement of low-incomeminority students by about 15ndash23 percentile points per year24

Given an in-sample standard deviation of about nineteen percen-tile points on the math test25 this suggests effect sizes on theorder of 008 s ndash012 s per year or 032 s ndash048 s over four yearswhich are quite large for education production functions (see forexample Greenwald Hedges and Laine 1996) On the otherhand the effects on the reading scores are as often negative aspositive and are nearly always statistically indistinguishablefrom zero

B The Relative Effect of the Choice Schools

Table VII shows structural estimates (from equation (6)) ofthe causal effect of choice schools on educational attainment

24 I have also tried excluding those students who leave the choice schoolsbecause they reached the terminal grade for the school and those who scored 5 orlower on the math tests And I have interacted the years since application with thestudentrsquos sex race income preapplication test score and grade at application Theresults in Table VI remain essentially unchanged

25 Nationally the standard deviation of a normal curve equivalent score is 21percentile points

FIGURE IIAdjusted Reading (NCE) Test Scores by Years Since Application to Choice

Program All CohortsCoefficient estimates from a regression of the reading scores on dummy

variables for years since application years since application interacted withwhether the student was selected to attend a choice school or whether thestudent was not selected to attend a choice school and individual xed-effects

QUARTERLY JOURNAL OF ECONOMICS584

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 33: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

These equations are basically similar to those estimated in TableVI although I divided selected applicants into those who wereactually enrolled in a choice school and those who were not I allowa different intercept for students enrolled in a choice school thosewho applied to a choice school but were not selected26 and thosewho were enrolled in a choice school at one time but had returnedto the Milwaukee public schools (ie they received partial treat-ment)27 The students in the Milwaukee public schools compari-son group are the omitted category These categories are mutuallyexclusive in any test year I also allow these groups to have

26 There are seven students who were not accepted and who were enrolled ina choice school nonetheless I include them in the category lsquolsquoenrolled in a choiceschoolrsquorsquo

27 I also include in this category the few applicants who were selected for achoice school and did not enroll

TABLE VIIINDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON THE

RATE OF GROWTH IN TEST SCORES USING THE MILWAUKEE PUBLIC SCHOOLS

SAMPLE AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Enrolled in choice school 2 3764(1191)

0315(1098)

Not selected for choice school (choiceapplicant control group)

2 2858(1749)

2 0660(1591)

Selected for choice not currentlyenrolled in a choice school

2 1432(2155)

2783(1987)

Enrolled in choice school 3 number ofyears since application

2379(0481)

2 0291(0438)

Not selected for choice 3 number ofyears since application

0437(0786)

0101(0709)

Selected for choice not currentlyenrolled in choice school 3 numberof years since application

1772(0756)

2 0616(0694)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed All regressions control for unrestricteddummy variables indicating the number of years before or after application

These students applied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 585

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 34: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

different yearly (postapplication) increases in test scores Bothspecications include individual xed-effects

The results in column (1) suggest that the math scores ofstudents in the choice schools increased an additional 24 percen-tile points per year and the effect is statistically signicant at the5 percent level Recall that the reduced-form estimates shouldroughly equal the causal effect of choice schools on studentachievement scaled by the take-up rate using the notation fromequation (2) p 1 5 r b In all years having been selected toparticipate in the choice program increases a studentrsquos likelihoodof actually attending a choice school by about 70 percentagepoints Therefore the causal estimate should be approximately14p 1 or 204 based on the coefficient on lsquolsquoselected 3 number ofyears since applicationrsquorsquo in column (2) of Table VIa The actualestimate of 238 is only slightly higher than lsquolsquoexpectedrsquorsquo Theresults for reading scores in column (2) suggest that students inthe choice schools scored lower than students in the Milwaukeepublic schools by about 2 03 percentile points per year althoughthis difference is not statistically signicant

Next I implement an instrumental variables (IV) strategy forwhich I estimate the following equation

(9) Tit 5 a 1 b 80Pit 1 b 81CPit 1 Xi g 1 Zi G 1 d git 1 e it

where the vectors Xi and Zi are dened as before git indicates thegrade level of the student in the year of the test e it is an errorterm Pit indicates whether a student is currently enrolled in achoice school and CPit measures the total number of years thestudent has continuously been enrolled in a choice school or hadever been enrolled in a choice school as of and including year tThis measure will capture the fact that a substantial number ofstudents leave or have interrupted spells in the choice schoolswhich is important since I do not have a second instrumentalvariable with which to estimate a separate intercept and slope forthose who enrolled in a choice school and later returned to theMilwaukee public schools28 I then use whether a student was(randomly) selected to attend a choice school and the number of

28 However note that because I do not have a second instrumental variableI cannot allow for separate effects of the choice schools for those with continuousenrollment and those with interrupted enrollment

QUARTERLY JOURNAL OF ECONOMICS586

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 35: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

years since application interacted with whether the student wasrandomly selected as instrumental variables29

The OLS estimates are reported in column (1) and IV resultsin column (2) of Table VIII The coefficient estimate suggests thatthe choice schools increased studentsrsquo math test scores an addi-tional three percentile points per year over students in theMilwaukee public schools a gain that is statistically signicantThis estimate however likely overstates the true effect of theprogram If the effect of the cumulative number of years in the

29 I have also estimated a specication that included whether the studentwas enrolled the number of years since application and an interaction betweenwhether the student was enrolled and the number of years since application I theninstrumented for whether the student was enrolled and the interaction term withwhether the student was randomly selected and an interaction between whetherthe student was randomly selected and the number of years since application Theresults are quite similar

TABLE VIIIOLS AND IV ESTIMATES OF THE EFFECT OF CHOICE SCHOOLS ON MATH AND

READING TEST SCORES

OLS IV

(1) (2)

Dependent variable 5 math (NCE) scores

Currently enrolled in a choiceschool

2 1873(1031)

2 1206(1305)

Cumulative number of yearsenrolled in a choice school

1825(0592)

2987(0866)

R2 0093Number of observations 3177 3177

Dependent variable 5 reading (NCE) scores

Currently enrolled in a choiceschool

0089(0935)

1947(1168)

Cumulative number of yearsenrolled in a choice school

0289(0545)

0131(0810)

R2 0065Number of observations 3163 3163

Huber standard errors (that allow for individua l correlation) are in parentheses These regressionsinclude a constant lsquolsquoapplicant poolrsquorsquo dummy variables a dummy variable for female family income anindicator if income is missing and the grade level of the student when she took the test The math scoreregressions include a dummy variable indicating if the test score was imputed The instruments are whetherthe student was randomly selected to attend a choice school and whether the student was randomly selectedinteracted with years since application

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 587

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 36: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

program has a different slope for those who are currently enrolledthan for those who had been enrolled but have returned to theMilwaukee public schools (if the effects of attending a choiceschool fade out over time once a student returns to the publicschools for example) then the IV estimate will overstate thecumulative effect of the program30 I continue to nd no effect forreading scores Overall the IV strategy generates estimates thatare consistent with those reported in Table VII

C Reconciling These Results with Witte 1997

My ndings on the reading scores agree with those reportedby Witte Sterr and Thorn 1995 and Witte 1997 although myresults for the math scores do not Recall that these paperscompare the achievement of students in the choice schools withthat of a random sample of students from the Milwaukee publicschools The primary difference between our analyses is thatWitte and his coauthors estimate a lsquolsquoquasi-gainrsquorsquo specication inwhich they include prior math and reading test scores as covari-ates whereas I include individual xed-effects

Witte and his coauthors estimate the following lsquolsquoquasi-gainrsquorsquospecication

(10) Tit 5 a 8 1 d 8Pit 1 g 8Yit 1 b 8(Yit 3 Pit ) 1 l Tit2 1 1 e 8it

where Tit2 1 is the lagged-test score I refer to this model as alsquolsquoquasi-gainrsquorsquo (rather than a lsquolsquogainrsquorsquo) specication because thelagged values of other covariates are not included In column (1) ofTable IX I report Wittersquos 1997 estimates of equation (10)31 Incolumn (2) I present my best attempt to replicate his coefficientestimates32 In addition at the bottom of the table I present theimplied cumulative difference in test scores between the studentsenrolled in a choice school and those in the Milwaukee publicschools33 My replication is close to reproducing his results Thesemodels indicate that students in the choice schools did not havestatistically signicant faster gains in math

30 See Rouse 1997 for a more complete discussion of this bias31 I attempt to replicate the results in Witte 1997 as these are the most

recent I thank John Witte for providing me with coefficient estimates andstandard errors not reported in his paper

32 See the Data Appendix for details about this sample33 The cumulative effects do not incorporate the effect of the lagged test

score However doing so does not substantially change the results

QUARTERLY JOURNAL OF ECONOMICS588

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 37: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

TABLE IXRECONCILING WITH WITTE 1997rsquoS MATH TEST SCORE ESTIMATES

Wittedagger Rouse replication of Witte

Specication type

Quasi-gain Quasi-gainFirst-

differencedDagger FE FE

(1) (2) (3) (4) (5)

Enrolled in a choiceschool

2 1525(1247)

2 1160(1199)

2 0924(1126)

0867(1226)

2 0816(1084)

Number of yearsbefore or afterapplication

2 0332(0222)

2 0350(0221)

2 1072(0173)

2 0972(0140)

Enrolled in a choiceschool 3 numberof years

0592(0523)

0560(0493)

0930(0525)

0787(0587)

1782(0436)

Lagged math score 0527(0016)

0528(0016)

Lagged readingscore

0160(0017)

0156(0017)

R2 0437 0439 0073 0799 0787Number of observa-

tions3967 3862 3862 5424 7797

Implied cumulative (differential) effect ofbeing enrolled in a choice school

First year 2 0932 2 0599 0006 1654 0966(0800) (0783) (0962) (0930) (0906)

Second year 2 0340 2 0039 0936 2440 2748(0523) (0523) (1065) (0956) (0922)

Third year 0252 0522 1866 3227 4529(0674) (0649) (1377) (1286) (1122)

Fourth year 0845 1082 2796 4013 6311(1088) (1026) (1792) (1755) (1431)

The dependent variable is the math (NCE) score Standard errors are in parentheses columns (1)ndash(3)report Huber standard errors that allow for correlations lsquolsquowithinrsquorsquo an individual Columns (1)ndash(3) also includethe grade level of the test and dummy variables for female African-American Hispanic and other minorityAll samples only include lsquolsquolow-incomersquorsquo students (eg qualied for a free or reduced lunch) The specicationsin column (1) and (2) correspond to equation (10) in the text and that in column (3) corresponds to equation(13) FE columns include individua l xed-effects

The sample only includes students who are not missing prior math and reading test scores The sample does not exclude students missing prior math or reading test scoresdagger Based on lsquolsquoTable II col 5SRDBrsquorsquoDagger This specication is fully rst-diffenced the right-hand-side variables are also rst-differences

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 589

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 38: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

One interpretation of the specication in equation (10) is thatit derives from a dynamic model in which the test score in period tis a direct function of the test score in period t 2 1 without anexplicit individual xed-effect Although the model does notexplicitly allow for an individual xed-effect equation (10) implic-itly controls for individual ability by including the lagged readingtest score among the covariates Another way to compare thequasi-gain and xed-effects specications is to see that with onlytwo periods of data the xed-effects and rst-differenced specica-tions are equivalent and that the rst-differenced is a nestedversion of the quasi-gain specication when the rst period is alsquolsquopreprogramrsquorsquo period To see this consider the following underlyingequation

(11) Tit 5 a t 1 d Pit 1 g Yit 1 b (Yit 3 Pit ) 1 v i 1 e it

where again Tit represents the test scores of child i in period t andv i is a time-invariant individual effect A similar equation de-scribes the determinants of test scores in the period t 2 1

(12) Tit 2 1 5 a t2 1 1 d Pit 2 1 1 g Yit2 1 1 b (Yit 2 1 3 Pit2 1) 1 v i 1 e it2 1

One can control for individual ability by substituting for v i inequation (11) using equation (12) The resulting equation is

(13) Tit 5 a 1 d Pit 2 Pit 2 1 1 g Yit 1 b (Yit 3 Pit )

2 (Yit 2 1 3 Pit2 1) 1 Tit2 1 1 e 8it

where e 8it 5 e it 2 e it2 1 and a 5 a t 2 a t 2 1The rst-differenced specication imposes the constraint that

the coefficient on the lagged-test score equals one and is thereforeequivalent to a xed-effects specication with only two periods ofdata Similarly when t 2 1 is a lsquolsquopreprogramrsquorsquo period such thatPit 2 1 5 Yit 2 1 5 0 and there are no additional time-varyingcovariates the rst-differenced specication is equivalent to thequasi-gain specication with the constraint that the coefficient onthe lagged test score equals one With additional years of data andtime-varying covariates the models are less similar Howeverbecause there are at most four periods of data (in this sample) anda limited number of covariates all three specications likely yield(roughly) similar parameter estimates

The coefficient estimates based on a rst-differenced specica-tion (in column (3)) are statistically similar to the quasi-gain

QUARTERLY JOURNAL OF ECONOMICS590

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 39: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

estimates when the same sample is used Hypothesis testsindicate that the estimate of the interaction between whether thechild is enrolled in a choice school and the number of years sinceapplication does not statistically differ from 056 the estimate incolumn (2) Similarly the estimates of the cumulative effects ofthe choice schools do not differ from those in column (2) Incontrast the xed-effects specication shown in column (4) ap-pears to generate similar estimates in some respects but notothers34 On the one hand the coefficient estimate on the interac-tion between being enrolled in a choice school and the number ofyears since application does not statistically differ from that incolumn (2) On the other hand the cumulative effect of beingenrolled in a choice school is statistically signicant in the secondthird and fourth years after application These results suggestthat my results differ from those presented by Witte partiallybecause of differences in our specications

There are some reasons to prefer the xed-effects specica-tion to the quasi-gain specication however First although thelagged test score provides a measure of unobserved ability thatvaries over time it is likely a noisy measure of unobserved abilitybecause of measurement error varying testing conditions andother unobserved inuences that affect student test scores in anygiven year35 Second referring to equation (13) one can see thatthe quasi-gain specication cannot be derived from a simple linearmodel of test scores without omitting the variable Pit 2 136 How-ever in the model underlying equation (13) this restriction is onlyvalid for preprogram years it is an omitted variable in this

34 The number of observations in the xed-effects (FE) columns is slightlyhigher than that in the OLS columns because I include the observations from thelsquolsquoyear of applicationrsquorsquo in order to identify the main effect of lsquolsquoenrolled in a choiceschoolrsquorsquo These observations are excluded from the gain and quasi-gain specica-tions (in this sample) because there is no prior test score for the year of application

35 Note as well that because of the relatively short time series anindividual xed-effects models will likely generate biased estimates of the effect ofthe choice schools because of the lagged dependent variable Hsiao 1986

36 As another approach one could determine the lsquolsquolevelrsquorsquo specication towhich the quasi-gain specication corresponds Because the effect of the choiceprogram is measured as a trend coefficients from the quasi-gain specicationcorrespond to coefficients from a quadratic specication in levels that includesindividual xed-effects When I estimate such a quadratic specication includingindividual xed-effects the coefficients largely correspond as expected Howeverthis implies that the lsquolsquolinearrsquorsquo effect of the program from the quasi-gain specicationis in fact the quadratic term

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 591

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 40: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

situation where most of the data are from periods when childrenhave been enrolled in choice schools for multiple years37

Finally the xed-effects specication has the advantage ofcontrolling for (time-invariant) individual ability using all avail-able data Specically in order to implement the quasi-gainspecication one must exclude all students who are missing alagged test score These missing data present a potentially largeproblem since 75 percent of the students in the Milwaukee publicschools sample and 59 percent of the choice student sample aremissing the prior test scores To evaluate whether the missingdata can explain the differences in our results I estimated thexed-effects specication (from column (4)) using the entiresample of students not just those who are not missing the priortest score The results are in column (5) The coefficient estimateof 18 and the implied cumulative effects are roughly similar tothose presented in Tables Va VI and VII suggesting that mymath results also differ from those reported by Witte because mypreferred xed-effects model takes advantage of the largersample38

V CONCLUSION

The results using the quasi-experimental applicant controlgroup and the random sample of students from the Milwaukeepublic schools as a comparison group (when I include individualxed-effects) are remarkably similar On the one hand I nd that

37 Krueger 1997 reaches a similar conclusion in his study of the achieve-ment effects of smaller class sizes using the Project STAR experiment He ndsthat most of the positive effect of being enrolled in a small class occurs in the rstyear Because of the panel nature of the study and since there are no lsquolsquopreprogramrsquorsquotest scores a quasi-gain specication would have substantially understated thepositive benets of the smaller class sizes I am currently investigating the implicitrestrictions imposed by alternative models of education production functions withpanel data

38 The results for reading do not appear as sensitive to the sample selectionI have also tried imputing the missing prior test scores but the results aresensitive to the method of imputation The increase in the effect of the program (incolumn (5)) does not appear to be only due to a larger treatment effect for thosestudents missing the prior test scores When the sample is restricted to thosemissing prior test scores the point estimate on the effect of the choice program is1230 with a standard error of 2008 when individual xed-effects are includedHowever note that the main effect of being in a choice program cannot be identiedwith this sample since there are no preapplication observations by constructionThe corresponding point estimate suppressing the main effect of being in a choiceprogram and using the sample excluding those missing prior test scores is 1070with a standard error of 0428

QUARTERLY JOURNAL OF ECONOMICS592

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 41: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

on average students selected for the Milwaukee Parental ChoiceProgram and those enrolled in the participating private schoolslikely scored 15ndash23 percentile points per year in math more thanstudents in the comparison groups On the other hand the resultsfor reading scores were quite mixed with both positive andnegative coefficient estimates

In addition I conclude that my results for the reading scoresdiffer from those reported by Greene et al primarily because theirestimates are not robust to the inclusion of individual xed-effectsand alternative specications My results for math differ fromthose reported by Witte partly because he excludes students withno lsquolsquolagged test scorersquorsquo in order to implement his specication andpartly because of our specications However the xed-effectsspecication has several advantages over the quasi-gain specica-tion particularly with these (panel) data in which students haveattended a choice school over multiple periods

Although these results obtain using a variety of estimationstrategies and samples there are at least three caveats to keep inmind First I had to impute the total math score for a substantialfraction of the Milwaukee public schools students Second andmost importantly these estimates are only unbiased as long as thepreapplication test score trajectories of those selected for thechoice program and those of the unsuccessful applicants andstudents in the Milwaukee public schools sample were similarand as long as the sample attrition was nonrandom (or at least notcorrelated with being enrolled in a choice school) While descrip-tive statistics and econometric techniques for addressing sampleattrition suggest that these are reasonable assumptions theremay be residual unobserved differences for which I cannot controlThird these are average effects that do not necessarily mean all ofthe choice schools are lsquolsquobetterrsquorsquo than the Milwaukee public schools

The data collection from Milwaukee should be applauded as itallows us to learn more about the effectiveness of this programthan from many other reforms Nevertheless if one lesson emergesfrom this effort it is that future evaluations of reforms of this sortshould anticipate high mobility among the students and recognizethat administrative data are determined by the needs of theschools and not those of the evaluator An evaluation design thattreats the participants (and control or comparison group) as asurvey sample with independent follow-up though more costlywould avoid some of the data problems experienced here

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 593

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 42: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

Finally although the experience in Milwaukee suggests thatproviding vouchers to low-income students to attend privateschools could help increase the mathematical achievement ofthose students who participate (on average) it cannot shed lighton whether vouchers provide an incentive for the public schools toimprove and therefore increase the quality of education providedto all low-income children In addition the results from oneprogram implemented in one city cannot and should not be theonly evidence on which important policy regarding the structureof American education is based It is only by piecing togetherevidence from many places that we will ever really learn whetherprivate school vouchers could increase student achievement

DATA APPENDIX

The data on the Milwaukee Parental Choice Program can bedownloaded from httpdplsdaccwisceduchoicechoice_in-dex_html I construct the analysis sample according to thefollowing denitions which are similar to those used by Greene etal 1996

Year of Application

I dene the year of application as the rst year in which astudent applied to attend a choice school if she either was neverselected or was selected the rst time she applied If the studentapplied more than once I consider the rst year she was acceptedas the year in which she applied In addition in a few cases thestudents applied were not selected but were nonetheless enrolledin a choice school the following spring (ie they were admitted offof a waiting list) I consider their year of application the year inwhich they applied and were not accepted as once students wereenrolled in a choice school they did not participate in therandomization the following years I only include those who rstapplied in the years 1990ndash1993

For students in the Milwaukee public schools sample Iconsider 1990 the lsquolsquoyear of applicationrsquorsquo for students in theMilwaukee public schools who have a valid 1991 test score Forthose without a 1991 test score I consider their lsquolsquoyear of applica-tionrsquorsquo to be 1991 if they have a valid 1992 test score and so forth

QUARTERLY JOURNAL OF ECONOMICS594

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 43: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

Grade at Application

I only use the grade levels of the Iowa Tests of Basic Skills indetermining the grade to which the student applied As the test isadministered in the spring semester I consider the grade level ofthe test of the spring following the year of application as the gradeto which the student applied To impute the grade of application(in cases in which the student is missing a test score for the rstspring following the year of application) I search backward andforward in the test data for a nonmissing test grade and add orsubtract the appropriate number of years For example supposethat a student applied in 1990 is missing a 1991 test score buthas a test score for the spring of 1992 I consider the grade atapplication to be the 1992 grade minus one (An alternativemethod for imputing would be to use the grade of the student fromthe administrative data (the mastchdat and chsrdbdat les))The exact order in which one searches for a valid grade using thetest score data and using the alternative method of imputationmakes almost no difference for the point estimates for the mathscores but makes a small difference for the point estimates forreading I only include students who applied to grades Kndash8 in theanalysis

Grade Level of Test

I primarily use the grade levels of the Iowa Tests of BasicSkills in determining the grade of the student at the time of thetest However there are six students in the Milwaukee publicschools sample for whom I lsquolsquoimputersquorsquo a grade level using informa-tion from the administrative data (ctsrdbdat) and from the testscore grade levels of other years

Race and Sex

I rst determine the race and sex of the student from themaster choice le (mastchdat) Further I require that the raceand sex of the student be consistent in all years (Thus if thestudent appears to lsquolsquochangersquorsquo sex or race across years I considerthe race or sex missing) If the race or sex is missing from theseles I use the race and sex from the choice Student RecordDatabase (SRDB) (chsrdbdat) le Again I require that the raceand sex not change over the data set I only include African-Americans and Hispanics in most of the analysis

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 595

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 44: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

Test Scores

I use the normal curve equivalent (NCE) transformation ofthe math and reading scores For both I recompute them based onthe transformation in Thorn Witte and Sterr 1995 as several ofthe test scores appear to have been mistransformed I only usecurrent test scores I impute math scores for students in theMilwaukee public schools who do not have a total score but dohave a score for the problem-solving component I impute usingthe following equations

mnce93 5 16293 1 0250mnpr93 1 0023mnpr932

2 000029mnpr933 1 00000013mnpr934

2 0079tsyear93

R2 5 0810 N 5 3603

mnce94 5 9778 1 0265mnpr94 1 0023mnce942

2 000031mnce943 1 00000014mnce944

2 0017tsyear94

R2 5 0814 N 5 3182

where mnce9X is the total math score mnpr9X is the problem-solving component and tsyear9X is the year the test was adminis-tered I estimate these equations using the random sample ofstudents in the Milwaukee public schools (and I include studentswith test scores from previous years)

The preapplication test score is the test score from thestudentrsquos lsquolsquoyear of applicationrsquorsquo

Family Income and Parental Education

Family income is from the survey administered to choiceapplicants in the fall and spring of each year as well as to arandom sample of families in the Milwaukee public schools Iaverage the reported income from the fall and spring of the yearafter application (to lower the measurement error) If a studentapplied more than once (and was not immediately selected) and ismissing information on family income I ll in the informationwith the reported income in earlier years I convert the measure to1994 dollars and substitute the average (conditional on whether

QUARTERLY JOURNAL OF ECONOMICS596

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 45: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

the student was ever-selected never-selected or part of theMilwaukee public schools sample) for those missing income

I construct parental education in a similar fashion I convertthe categorical variable to a continuous education measure basedon the mapping suggested by Park 1994 8th grade or below 5 8some high school 5 10 GED and high school graduate 5 12 somecollege 5 13 four-year degree 5 16 and postgraduate work 5 17

Choice Students

I consider an individual a choice student who attendedprivate school if she or he took an achievement test in a choiceschool In the analysis I exclude a few students who only had validtest scores from a Milwaukee public school but who were indicatedas enrolled in a choice school in the same spring There are also sixstudents who appear to be enrolled in a choice school in aparticular spring and also appear to apply for the rst time in thesame spring and there are two students who apply in 1990 or1991 only appear to be enrolled in a choice school in 1993 andwho have test scores from both the choice school and a Milwaukeepublic school in 1993 I include these students although theresults are quite similar when I exclude them

My Replication of Wittersquos Sample

This sample includes all low-income students (not onlyAfrican-Americans and Hispanics) in the Milwaukee public schoolssample and those enrolled in the choice schools Low-incomestudents are those who qualied for a free- or reduced-lunch allchoice students are dened as lsquolsquolow-incomersquorsquo students lsquolsquoChoicersquorsquostudents are dened as those who were accepted by enrolled in ortook a test in a choice school Unsuccessful applicants who neverenrolled in a choice school and choice students once they leave thechoice school are excluded from the analysis

The test scores have not been retransformed from percentileto normal curve equivalents and I use Wittersquos 1997 imputationequation for the total math scores When constructing the lsquolsquopriortest scoresrsquorsquo I do not rst exclude individuals with missing testscores in a particular year Thus the prior test score is moreaccurately lsquolsquolast yearrsquos test scorersquorsquo The results are not sensitive tothis decision

The lsquolsquoyear of applicationrsquorsquo is set to 1990 for all students in theMilwaukee public schools These estimates only use test scoresbeginning in the lsquolsquoyear of applicationrsquorsquo

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 597

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 46: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

APPENDIX 1 SAMPLE MEANS AND STANDARD DEVIATIONS

Selected Not-selected MPS

Proportion currently enrolled in achoice school

05580497

00100102

NA

Math (NCE) score 3923618706

3810718777

4041118483

Reading (NCE) score 3764716265

3788617089

3870016461

Proportion female 0536 0470 05230499 0499 0499

Family income ( 4 1000) (1994 dollars) 120525915

125105901

217508658

Proportion missing family income 04120492

05420498

07470435

Proportion African-American 07950403

08640343

08680338

Proportion Hispanic 0205 0136 01320403 0343 0338

Grade level of test 37672250

38572088

43372122

Proportion with imputed math testscore

01550363

05270500

04890500

Proportion applied in 1990 03640481

03060461

07530431

Proportion applied in 1991 03310471

01790384

01760381

Proportion applied in 1992 01650371

03240468

00600238

Proportion applied in 1993 01400347

01910393

00100100

Proportion with test score in 1990 01170322

01580365

01750380

Proportion with test score in 1991 01680374

02060405

02490432

Proportion with test score in 1992 02370425

02670442

02400427

Proportion with test score in 1993 02280419

02180413

01840388

Proportion with test score in 1994 02500433

01510359

01520359

Number of observations 2462 859 5408

Standard deviations are in brackets Based on the sample for math scores lsquolsquoMPSrsquorsquo is an abbreviation forlsquolsquoMilwaukee public schoolsrsquorsquo sample This sample includes multiple observations per student (it is the fulllsquolsquopanelrsquorsquo)

QUARTERLY JOURNAL OF ECONOMICS598

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 47: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

APPENDIX 2 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH AND READING SCORES USING THE RANDOM

SAMPLE OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable

Math (NCE)scores

Reading (NCE)scores

(1) (2)

Three years before application 2332 2803(1754) (1603)

Two years before application 3557 2512(1057) (0956)

One year before application 3615 2312(0809) (0731)

One year after application 0766 0031(0519) (0477)

Two years after application 2 0083 2 0504(0575) (0525)

Three years after application 2 2863 2 1476(0660) (0576)

Four years after application 2 4090 2 2738(0787) (0656)

Selected to attend a choice school 3 one yearafter application

2 0740(0941)

0649(0868)

Selected to attend a choice school 3 twoyears after application

0301(1027)

2 0222(0945)

Selected to attend a choice school 3 threeyears after application

3863(1154)

0498(1057)

Selected to attend a choice school 3 fouryears after application

6191(1503)

2 0559(1379)

Not selected to attend a choice school 3 oneyear after application

2 2733(1357)

2 0255(1240)

Not selected to attend a choice school 3 twoyears after application

2 1191(1573)

2 0268(1436)

Not selected to attend a choice school 3 threeyears after application

0214(2123)

2 1194(1924)

Not selected to attend a choice school 3 fouryears after application

2 2457(2710)

1902(2487)

Constant 39624 38357(0361) (0327)

R2 0761 0738Number of observations 8729 8751

Standard errors are in parentheses The estimates include individua l xed-effects and for the math testscores a dummy variable indicating that the total score was imputed These are the coefficients and standarderrors underlyin g the estimates in Figures I and II lsquolsquoNot selected to attend a choice schoolrsquorsquo are students whoapplied to the choice program but were not accepted

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 599

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 48: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

PRINCETON UNIVERSITY AND NATIONAL BUREAU OF ECONOMIC RESEARCH

APPENDIX 3 INDIVIDUAL FIXED-EFFECTS ESTIMATES OF THE EFFECT OF SELECTION

TO THE CHOICE PROGRAM ON MATH SCORES USING THE RANDOM SAMPLE

OF MILWAUKEE PUBLIC SCHOOLS STUDENTS AS A COMPARISON GROUP

Dependent variable 5 math(NCE) scores

1991ndash1993cohorts only

Excludes 1994test scores for1990 cohort

(1) (2)

Three years before application 2310 2323(1753) (1752)

Two years before application 3575 3548(1066) (1056)

One year before application 3687 3609(0829) (0809)

One year after application 0771 0765(0518) (0519)

Two years after application 2 0061 2 0083(0573) (0575)

Three years after application 2 2788 2 2860(0661) (0661)

Four years after application 2 3974 2 4084(0792) (0790)

Selected to attend a choice school 3 one yearafter application

2 1043(1091)

2 0774(0943)

Selected to attend a choice school 3 twoyears after application

1027(1230)

0174(1031)

Selected to attend a choice school 3 threeyears after application

5112(1449)

3844(1160)

Not selected to attend a choice school 3 oneyear after application

2 2508(1537)

2 2727(1361)

Not selected to attend a choice school 3 twoyears after application

2 0978(1887)

2 1040(1585)

Not selected to attend a choice school 3 threeyears after application

1365(3626)

1121(2160)

Constant 39820 39653(0387) (0361)

R2 0766 0765Number of observations 7570 8548

Standard errors are in parentheses The estimates include individua l xed-effects and a dummy variableindicating that the total score was imputed lsquolsquoNot selected to attend a choice schoolrsquorsquo are students who appliedto the choice program but were not accepted

These sample restrictions only apply to the applicants

QUARTERLY JOURNAL OF ECONOMICS600

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 49: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

REFERENCES

Angrist J G Imbens and D Rubin lsquolsquoIdentication of Causal Effects UsingInstrumental Variablesrsquorsquo Journal of the American Statistical Association XLI(1996) 444ndash455

Ashenfelter Orley lsquolsquoThe Effect of Manpower Training on Earnings PreliminaryResultsrsquorsquo in Proceedings from the Twenty-Seventh Annual Winter Meeting ofthe Industrial Relations Research Association (Madison WI Industrial Rela-tions Research Association 1975)

Cain Glen G and Goldberger Arthur S lsquolsquoPublic and Private Schools RevisitedrsquorsquoSociology of Education LVI (1983) 208ndash218

Coleman James Thomas Hoffer and Sally Kilgore High School AchievementPublic Catholic and Private Schools Compared (New York Basic Books1982a)

Coleman James Thomas Hoffer and Sally Kilgore lsquolsquoCognitive Outcomes in Publicand Private Schoolsrsquorsquo Sociology of Education LV (1982b) 65ndash76

Cookson Peter W lsquolsquoAssessing Private School Effects Implications for SchoolChoicersquorsquo in School Choice Examining the Evidence Edith Rasell and RichardRothstein eds (Washington DC Economic Policy Institute 1993)

Coronary Drug Project Research Group lsquolsquoInuence of Adherence to Treatment andResponse to Cholesterol on Mortality in the Coronary Drug Projectrsquorsquo NewEngland Journal of Medicine XXXIII (1980) 1038ndash1041

Efron B and D Feldman lsquolsquoCompliance as an Explanatory Variable in ClinicalTrialsrsquorsquo Journal of the American Statistical Association LXXXVI (1991) 9ndash17

Epple Dennis and Richard E Romano lsquolsquoEnds against the Middle DeterminingPublic Service Provision When There Are Private Alternativesrsquorsquo Journal ofPublic Economics LXII (1996) 297ndash325

Evans William N and Robert M Schwab lsquolsquoFinishing High School and StartingCollege Do Catholic Schools Make a Differencersquorsquo Quarterly Journal ofEconomics CX (1995) 941ndash974

Figlio David N and Joe A Stone lsquolsquoSchool Choice and Student Performance ArePrivate Schools Really Betterrsquorsquo University of Oregon mimeo April 1997

Goldberger Arthur S and Glen G Cain lsquolsquoThe Causal Analysis of CognitiveOutcomes in the Coleman Hoffer and Kilgore Reportrsquorsquo Sociology of EducationLV (1982) 103ndash122

Greene Jay P Paul E Peterson and Jiangtao Du lsquolsquoThe Effectiveness of SchoolChoice The Milwaukee Experimentrsquorsquo Harvard University Education Policyand Governance Occasional Paper 97-1 March 1997

Greene Jay P Paul E Peterson Jiangtao Du Leesa Boeger and Curtis L FrazierlsquolsquoThe Effectiveness of School Choice in Milwaukee A Secondary Analysis ofData from the Programrsquos Evaluationrsquorsquo University of Houston mimeo August1996

Greenwald Rob Larry V Hedges and Richard D Laine lsquolsquoThe Effect of SchoolResources on Student Achievementrsquorsquo Review of Educational Research LXVI(1996) 361ndash396

Hausman Jerry A and David A Wise lsquolsquoTechnical Problems in Social Experimen-tation Cost versus Ease of Analysisrsquorsquo in Social Experimentation Jerry AHausman and David A Wise eds (Chicago IL The University of ChicagoPress 1985)

Heckman James and Jeffrey Smith lsquolsquoAssessing the Case for RandomizedEvaluation of Social Programsrsquorsquo in Measuring Labour Market MeasuresEvaluating the Effects of Active Labour Market Policies Karsten Jensen andPer Kongshoj eds (Copenhagen Ministry of Labour 1993)

Heckman James and Jeffrey Smith lsquolsquoSubstitution Bias in Social Experiments AnAnalysis of JTPA Datarsquorsquo University of Chicago mimeo 1994

Hoxby Caroline Minter lsquolsquoThe Effects of Private School Vouchers on Schools andStudentsrsquorsquo in Holding Schools Accountable Performance-Based Reform inEducation Helen F Ladd editor (Washington DC The Brookings Institution1996)

Hsiao Cheng Analysis of Panel Data (Cambridge Cambridge University Press1986)

Krueger Alan B lsquolsquoExperimental Estimates of Education Production FunctionsrsquorsquoNBER Working Paper No 6051 June 1997

PRIVATE SCHOOL VOUCHERS AND ACHIEVEMENT 601

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602

Page 50: PRIVATE SCHOOL VOUCHERS AND STUDENT ACHIEVEMENT: AN ...piketty.pse.ens.fr/files/Rouse1998.pdf · TABLEI MEANCHARACTERISTICSOFAPPLICANTSTOTHECHOICEPROGRAMANDSTUDENTSIN THEMILWAUKEEPUBLICSCHOOLS

Murnane Richard J lsquolsquoA Review EssaymdashComparisons of Public and PrivateSchools Lessons from the Uproarrsquorsquo Journal of Human Resources XIX (1984)263ndash277

Neal Derek lsquolsquoThe Effects of Catholic Secondary Schooling on Educational Achieve-mentrsquorsquo Journal of Labor Economics XV (1997) 98ndash123

Nechyba Thomas J lsquolsquoPublic School Finance in a General Equilibrium TieboutWorld Equalization Programs Peer Effects and Private School VouchersrsquorsquoNBER Working Paper No 5642 June 1996

Park Jin Heum lsquolsquoEstimation of Sheepskin Effects and Returns to Schooling Usingthe Old and the New CPS Measures of Educational Attainmentrsquorsquo IndustrialRelations Section Working Paper No 338 December 1994

Rouse Cecilia Elena lsquolsquoPrivate School Vouchers and Student Achievement AnEvaluation of the Milwaukee Parental Choice Programrsquorsquo NBER Paper No5964 March 1997

Rubin D lsquolsquoEstimating Causal Effects of Treatments in Randomized and Non-randomized Studiesrsquorsquo Journal of Educational Psychology LXVI (1974) 688ndash701

Sander William lsquolsquoCatholic Grade Schools and Academic Achievementrsquorsquo Journal ofHuman Resources XXXI (1996) 540ndash548

Thorn Christopher A John F Witte and Troy D Sterr lsquolsquoDocumentation of DataUsed in the Milwaukee Choice Studyrsquorsquo La Follette Institute of Public Affairsmimeo August 1995

Witte John F lsquolsquoPrivate School vs Public School Achievement Are There FindingsThat Should Affect the Educational Choice Debatersquorsquo Economics of EducationReview XI (1992) 371ndash394 lsquolsquoAchievement Effects of the Milwaukee Voucher Programrsquorsquo University ofWisconsin at Madison mimeo January 1997

Witte John F Troy D Sterr and Christopher A Thorn lsquolsquoFifth-Year ReportMilwaukee Parental Choice Programrsquorsquo University of Wisconsin mimeo Decem-ber 1995

Witte John F and Christopher A Thorn lsquolsquoThe Milwaukee Parental ChoiceProgram 19901991ndash19941995 computer lersquorsquo Madison WI 1995

Witte John F Christopher A Thorn lsquolsquoWho Chooses Voucher and InterdistrictChoice Programs in Milwaukeersquorsquo American Journal of Education CIV (1996)186ndash217

Witte John F Christopher A Thorn and Kim A Pritchard lsquolsquoPrivate and PublicEducation in Wisconsin Implications for the Choice Debatersquorsquo University ofWisconsin mimeo 1995

Witte John F Christopher A Thorn Kim M Pritchard and Michele ClaibournlsquolsquoFourth-Year Report Milwaukee Parental Choice Programrsquorsquo University ofWisconsin mimeo December 1994

QUARTERLY JOURNAL OF ECONOMICS602