Performance of School MatchingAlgorithms

Omar Hossain AhsanAdvised by Professor Peter Cramton

University of Maryland-College Park

oahsan@umd.edu

Abstract

A school match, or lottery, is a system in which students are matched to schools through aprocess in which they list their most preferred schools and are then matched to one of thoseschools using an algorithm. Two algorithms that have been studied in the past are a DeferredAcceptance model used in New York [1] and a Priority Matching mechanism used in Boston [2].In this paper we study the performance of these school matching algorithms in Washington,D.C. and the factors that influence their success. In particular, we focus on the number ofpreferences listed by students as well as different ways they put relative importance on theacademic quality of schools and their distance from the schools. We create a program thatgenerates student data and preferences based on Washington D.C.’s population distribution.The schools in the program are identical to the ones that participated in the D.C. school match.The program creates a score for each school for each student that then is used to generate thepreferences for every student. Since we generate preferences based on cardinal scores, we haveinformation on not just which schools a student prefers, but how much they prefer each schoolover the next best one. We study the performance of the two algorithms by comparing thedistribution of student utilities as the number of choices listed varies. We find that the NewYork algorithm achieves higher student utility on average than the Boston algorithm whenstudents list a small number of schools in their preferences and that the Boston algorithmperforms better when they list a larger number of schools.

1 Introduction

In a traditional school system, students are as-signed to schools based on where they live.Every school has a designated zone, whichis roughly the neighborhoods for which thatschool is the nearest school. Students are thenrequired to attend the school whose zone theyreside in. One issue with this system is that itdoes not allow students any possibility of choos-ing their own schools if their address is taken asfixed. To some degree this system does allow forschool choice if people are free to live in any partof the district they choose. Then parents couldsimply choose to live in a neighborhood that isassigned to the school they want their childrento attend . However, it is generally only wealthypeople that have the freedom to live in any partof the county that they choose. The quality ofthe schools is usually public information, whichleads to the corresponding neighborhoods hav-ing higher housing prices.

Because traditional school systems only al-low the richer families to have a choice in theschools they send their kids to, some school sys-tems have changed to school matches. A schoolmatch (or lottery) is a system in which studentsare assigned to schools based on their prioritiesat the schools and their own preferences. A stu-dent’s priority at a school can be thought of asthe rank the school assigns that student. In thispaper we take the priorities of the students ateach school as randomly determined. We as-sume that students will decide which schoolsthey prefer by their academic quality and theschools’ distance from their home. Students arethen matched to the schools using a matchingalgorithm that uses the above information.

Two examples of matching algorithms are aDeferred Acceptance algorithm, which is in usein New York City [1], and a Priority Match-ing algorithm, which is used in Boston [2]. Thetwo algorithms differ in that the Deferred Ac-ceptance algorithm only issues one acceptance

1

by a school to each student. It prioritizes thestudents that have a higher priority number ateach school. It allows for students who matchedwith a school early (so, as one of their higherchoices) to be displaced by another student whopreferred the school less but had a higher pri-ority at it. This algorithm is “strategy-proof,”meaning that the dominant strategy for stu-dents is to list their true preferences instead ofdeciding to list a school they like less but thinkthey are more likely to receive near the top asa “safety school.” The Boston algorithm is dif-ferent from this in that students may receivemultiple offers from different schools. Addition-ally, students who rank a school higher on theirpreferences cannot be displaced by other stu-dents who had higher priorities but ranked theschool lower. Unlike the Deferred Acceptancealgorithm, this algorithm is not strategy proof.Students are advised by Boston Public Schoolsto consider ranking a less popular school nearthe top of their preferences in order to bettertheir chances of getting a match.

In this paper we examine the performance ofthese school matching algorithms in Washing-ton D.C. We evaluate the performance of thesealgorithms by comparing the utility of each stu-dent when matched to a school using each algo-rithm. In 2014, Washington held its first publicschool match using an algorithm similar to theDeferred Acceptance algorithm in New York.71 percent of students were matched to one oftheir choices in the first year.

1.1 Literature Review

The merits of a school system with school choiceover a school system where students do not geta choice in the school they attend are studiedin two papers by Dr. Caroline Hoxby. Hoxbyargues that school choice will lead to net in-creases in school productivity since less produc-tive schools will see their students begin to leavefor the more productive schools since they areable to have some degree of choice in their en-rollments [4]. Hoxby cites three examples inMilwaukee, Michigan and Arizona where thecompetition posed by charter schools which al-lowed students to choose other schools lead to aboost in the productivity seen in public schools.The theoretical base for this increase in produc-

tivity in public schools is explained as the pub-lic schools get their funding from local propertytaxes. If parents are able to choose betweenschool districts they will favor ones that aremore productive at equivalent levels of prop-erty taxes. Therefore if the productivity in thepublic schools in an area are low they will alsosee a loss in funding over time as parents choosenot to live in the area.

Hoxby also studies the benefits to studentachievement, though first arguing that look-ing at student achievement is wrong-headedbecause studies are usually not able to studystudent achievement while holding school re-sources constant. To study gains in achieve-ment, Hoxby looks at the specific example ofEdison schools. Edison is a for-profit school-ing company that is often hired to turn aroundfailing schools, often in very low income ar-eas. Hoxby uses test score data and finds thatEdison students are 2.1 percent more likelyto be judged proficient on their states examswhen compared to their best match from pub-lic schools in the same district. Hoxby findsthe best match by using a measure called theselected average treatment effect from anotherpaper to find which public school was closestto the Edison school in this measure. The in-crease in achievement by attending these Edi-son schools provides some evidence that maysuggest that giving more students access toschools like Edison by implementing schoolchoice mechanisms may result in increases instudent achievement [5].

Matching algorithms also have applicationsoutside of matching students to high schools.Another matching mechanism can be found inmatching recent medical school graduates toresidency programs [6]. The paper talks abouthow early matching mechanisms in matchingstudents to residencies in the United Kingdomsuffered from stability issues. That is, situ-ations where a student and a hospital wouldboth prefer to be matched to each other ratherthan their current partners as produced by thematching. The question was asked if therecould be an algorithm that not only produced astable matching, but one that would make thedominant strategy of people participating in thematch the list their true preferences, without

2

any sort of safety schools. [6]

2 Methodology

In order to study the performance of thesematching algorithms in Washington, D.C.,we create a computer program to generate adataset based on the characteristics of Wash-ington D.C. We then run the two matchingalgorithms on the district to compare the per-formance of the two algorithms. In the DeferredAcceptance algorithm (used in New York andWashington D.C.), students are matched toschools as follows [1][3]:

Deferred Acceptance Algorithm (NewYork Algorithm)1. Each student applies to their top choiceamongst the schools they have not been re-jected from. This means that if in the previousround of the algorithm the student was notrejected from the last school they applied to,they do not apply to a different school in thisround.2. Schools consider the set of students thathave applied and haven’t been rejected. Thestudents are ordered by priority number. Theschool holds the top n students where n is thecapacity of the school. If more than n studentshave applied in this round then the rest of thestudents are issued rejections.3. Steps 1 and 2 repeat until either all studentsare matched to a school or there are no schoolsleft for any student.

In the Priority Matching algorithm, stu-dents are matched using the following steps [2]:

Priority Matching Algorithm (Boston Al-gorithm)1. Each student applies to their ith choiceschool in round i, where i starts at 1.2. Schools consider the students who have ap-plied in this round. The students are orderedby priority number and are assigned to theremaining seats in order until there are no ap-plicants or seats left. Any remaining applicantsare rejected.3. Steps 1 and 2 repeat until there are noschools left for any student.

One key difference is that in the Boston al-gorithm, acceptances are permanent and stu-dents in later rounds cannot displace previouslyaccepted students even if the later round stu-dent has a higher priority number.

2.1 Creating The Washington,D.C. Dataset

To create a dataset to represent Washington,D.C., we look at census tract data from the 2010and a map of the census tracts. The key piecesof information we use from each district are itspopulation, the percentage of the district thatis school-age children, latitude and longitude.We then use this information to generate theappropriate number of students for each cen-sus tract area, locating them all at the censustracts given latitude and longitude. The num-ber of students for each district is also impactedby the fraction of students in the tract thatchoose to attend private school instead of D.C.Public Schools (DCPS). In order to create theschools, we take the capacities and locations ofeach school as given by their capacities speci-fied by DCPS and their locations given by theirlatitudes and longitudes.

2.2 Generating Student Prefer-ences

In this model students generate preferences onschools based on the rankings and the locationsof the schools. Students create an academicscore for each school based on the schools per-formance on a countywide exam. Specificallyfor student i and school j, student i’s academicscore of school j is given by

aScorei,j =testScorej

maxk{testScorek}for all schools k.

Students also create a distance score foreach school which is based on the location ofthe school, the location of the student and thesize of the county. For student i and school j,and length of the county given by D, studenti’s distance score of school j is given by

dScorei,j = 1− ‖locationi − locationj‖D

3

The distance score is subtracted from 1 in orderto have shorter distances be mapped to higherscores. The student’s final score is created by

scorei,j = aScorei,j ∗ α+ dScorei,j ∗ β + ε

where α > 0, β > 0, α+ β = 1. ε is a randomterm that is uniformly distributed between 0and 1. The values of α and β are given inthree different scenarios for the district. Sce-nario one is “typical” scenario, where studentsmostly care about the academic scores of theschools but also factor in distance score as wellsince they prefer shorter commutes. In this sce-nario they put 75% of the weight on academics(α = 0.75) and 25% on distance (β = 0.25).Scenario two is used to represent a situationwhere the schools are all perceived as being ofsimilar quality and allows for students to putadditional value on distance with α = β = 0.5.Scenario three is used to represent a situationwhere travel costs are largely negligble rela-tive to the difference in academic scores of theschools and sets α = 0.9 and β = 0.1.

In the New York model, students then gen-erate their preferences simply based on thescores of each school i.e. if school i is scoredhigher than school j, then school i is rankedhigher than school j. However in the Bostonmodel, preferences cannot be generated thissimply, as simply stating true preferences is notnecesarily the best strategy for each student.Students in the Boston model will also take intoaccount their perceived chances of getting intoeach school when generating their preferences.To generate the chances, we run the Boston al-gorithm four times and calculate the acceptancerates for the schools each time. We then run thealgorithm one final time where students basetheir expectations on acceptance rates for eachschool based on data from the previous runs.Letting xi,j represent the acceptance rate forschool i in year j we calculate

xi,j = 0.2 ∗ xi,j−3 + 0.3 ∗ xi,j−2 + 0.5 ∗ xi,j−1

Using this information, for the Boston algo-rithm students generate their preferences basedon the score of the school and the schools ac-ceptance rate. For student a, school i is rankedbefore school j if

scorea,i ∗ acceptRatei ≥ scorea,j ∗ acceptRatej

The matching algorithms are then evaluatedbased on the percentage of students that re-ceived their top choice and the percentage ofstudents that do not receive any of their choices.Additionally we evaluate algorithms based onthe average score of the school each student re-ceives based on their preferences. We run thesetests on the district representing Washington,D.C. intially studying the effect of the numberof choices given by each student when using thefirst scenario (α = 0.75, β = 0.25) and then alsoon the other scenarios.

3 Results

3.1 Effect of Number of Choiceson Performance After OneRound

In this section we study the performance of theNew York and Boston algorithms on D.C. dataafter one round. After one round, not all stu-dents are guaranteed to be matched to a school.A student’s utility is given by:

ui =

{scorei,j if student imatched to school j0 if student i is unmatched

The simuaions run for section 3 are all based onthe first scenario for α and β described above.

Figure 1: Performance of New York algorithm

The New York algorithm sees large increasesin performance when increasing the number ofchoices listed by each student. The marginalincrease, however, decreases at high numbers ofchoices. This is not surprising as with a smallnumber of choices, many students are being left

4

unmatched. As students list more choices theirodds of receiving one increase. Since studentscannot be matched to more than one choicewe also do not have some students taking upmultiple spots so more students can be given amatch.

Figure 2: Performance of Boston algorithm

Increases in student utility as the numberof choices increases are smaller in the Bostonalgorithm. One of the differences between thetwo is that the Boston algorithm allows stu-dents to be matched to multiple schools. Thismeans that in the context of one round, manymore students may be left unmatched as schoolsappear full. Though the schools will later beopen for more applications the result is thatafter one round we see far more students leftunmatched. This can be seen in the followinghistogram showing the distribution of studentutilities when students are given 11 choices un-der both algorithms.

Figure 3: Distributions of utilities under both al-gorithms with 11 choices per student

As seen in the figure, the utilities amongststudents that have been matched is often higherunder the Boston algorithm than the New Yorkalgorithm. The New York algorithm, however,leaves far fewer students unmatched.

Figure 4: Comparison of Boston and New Yorkalgorithms - the y-axis is the probabilitythat a random utility from the Bostondistribution of utilities is greater than arandom utility from the New York dis-tribution

For a number of choices between 3 and 21we observe that a random utility taken from theNew York algorithm is expected to be higherthan one taken from the Boston algorithm,Boston is expected to be better for number ofchoices equal to 1 or 2. However at both ends ofthe graph we find that the probability is closeto 0.5 and that even in the center it remainsrelatively close to even. This contrasts fromlooking at the averages in the previous graphswhere the New York algorithm’s average utilityis significantly higher at these points. This fitswith the distribution seen in Figure 3 showingthat while the Boston algorithm often performsbetter amongst matched students, it still leavesmany unmatched after one round which is low-ering the average utility of the students.

3.2 Effect of Number of Choiceson Final Performance

In this section we continue running the algo-rithms for multiple rounds until all studentsare matched to a school. After each iterationof the algorithm, students who were matchedto a school are matched to that school (or, in

5

the Boston algorithm simulation, the top schoolamongst the ones they were matched to) andare removed from the simulations. Any fullschools are removed and the other schools havetheir capacities reduced appropriately. Thesimulations are then rerun with the remainingstudents.

Figure 5: Performance of New York algorithm

With the New York algorithm we see a dif-ferent effect of increasing the number of choiceson average student utility. When just looking atperformance after one round we saw utility in-crease as the number of choices by each studentgiven increased. This is largely due to far fewerstudents being left unmatched, rather than stu-dents getting more preferred choices. Whenlooking at final performance the algorithm isre-run and no students are left unmatched re-gardless of the number of schools listed by eachstudent. Thus this positive impact on the av-erage utility is lost. However by increasing thenumber of schools on students’ preferences weare actually increasing the probability that astudent who listed a school as their top choicegets displaced by another student who liked theschool less but had a higher priority number.In this case the decrease in utility may be ex-plained by a student with higher preferences forthe school being displaced by another studentwho actually does not prefer the school thatmuch.

Figure 6: Performance of Boston algorithm

We see that the Boston algorithm’s perfor-mance is relatively constant (compared to thechanges seen in the New York algorithm) evenas the number of schools allowed on students’preferences changes. There is just a small in-crease in average student utility as the numberof choices increases.

Figure 7: Comparison of Boston and New Yorkalgorithms

In this graph we see the trend between num-ber of choices given by students and the prob-ability that a random utility taken from theBoston distribution is higher than a randomutility taken from the New York distribution.For number of choices between 1 and 10 we seethat the New York algorithm achieves a bet-ter final result and that the Boston algorithmachieves a better final result for higher numberof choices.

When deciding how many schools to allowstudents to list on their preferences we find thatin the Boston algorithm the decision is rather

6

simple. Increasing the number of choices isgood because it improves the end result by asmall amount and will lead to algorithm to fin-ish faster. Increasing the number of choices be-yond a certain point however may not be prac-tical in real scenarios, as students may not thor-ough enough in ranking the schools to list thatmany choices at once.

For the New York algorithm districts shouldconsider whether they want to have a better re-sult after the first round, or have a better finalresult, as a smaller number will lead to a betterfinal result but a worse first round. Althoughit might seem better to maximize the final util-ity for all students after the completion of thematching program, there may be justificationfor maximizing the utility after round 1 as well.Even if the end result is better, students maynot be satisfied if a large number of them aretold that they were unmatched after the firstround and will have to go through the lotteryagain with fewer available spots in schools re-maining.

In an effort to find an “optimal” number ofchoices for students when using the New Yorkalgorithm, we look at the marginal effect of in-creasing the number of choices on both the per-formance after one round and the final perfor-mance after all students are matched.

Figure 8: Effect of increasing number of choiceson performance of the New York algo-rithm

The marginal benefit and marginal costcurves first intersect between 11 and 12 choices.Beyond that the cost curve is mostly above thebenefit curve but they intersect again and thedifference is small. For the D.C. district it

is then good to have students list at least 11choices because for numbers of choices between1 and 10 the marginal cost for the end resultis significantly below the marginal benefit forthe result after one round. At low numbers ofchoices the benefits to increasing the number ofchoices outweigh the costs.

4 Simulations With Differ-ent Weights for Aca-demics and Distance

As in the previous simulations, students scoreschools by computing:

scorei,j = aScorei,j ∗ α+ dScorei,j ∗ β + ε

In this section we study how the perfor-mance of the two algorithms changes whenthe weights for academic quality and distancechange. The above figures were all generatedwith the assumption that students weigh aca-demic quality 75% (α = 0.75) and distance 25%(β = 0.25). Here we consider two alternatives,α = 0.5 (scenario two) and α = 0.9 (scenariothree). For these sections we always run simu-lations to completion and analyze performanceafter all students have been matched.

4.1 Scenario Two

In this section we consider a scenario wherestudents perceive the academic quality of theschools as relatively similar and do not careso much about how much better one school isacademically than another. So we increase βwhile maintaining α+β = 1 and use the valuesα = β = 0.5. Students here weigh the distanceto schools and academics equally when makingtheir choices.

7

Figure 9: Performance of New York algorithm

Figure 10: Performance of Boston algorithm

We observe the same relationship betweennumber of choices and average utility for bothalgorithms that we did in scenario one. Themain difference is that utilities are overallhigher in scenario two. This is not surprisingas if students are more likely to prefer schoolscloser to them then students’ preferences willbe more varied and will lead to more studentsbeing given their top choices.

Figure 11: Comparison of Boston and New Yorkalgorithms

Overall we see the same trend as in scenarioone - that a random utility from the Bostonalgorithm is more likely to be higher than arandom utility from the New York algorithmas the number of choices listed goes up. How-ever, compared to scenario one, we see thatthis probability increases faster at first. TheBoston algorithm performs better for all num-ber of choices greater than 7 (and also at 5)compared to all number of choices greater than11 in scenario one.

4.2 Scenario Three

In this section we consider a scenario where stu-dents consider travel cost to be relatively negli-gible relative to the differences between schoolsacademically. A scenario for this might be ahighly dense county that subsidizes transporta-tion costs for students. Here we take α = 0.9and β = 0.1. Students consider academics to befar more important than distance when makingpreferences in this scenario.

8

Figure 12: Performance of New York algorithm

Figure 13: Performance of Boston algorithm

We again observe the same relationship be-tween number of choices and average utility forboth algorithms that we did in scenario one. Wealso expect that utilities are lower in this sce-nario. If students care mostly about academicquality of schools over distance then students’preferences will be more similar and there willbe more competition for the same schools. Theutilities at higher numbers of choices are lower,but the difference is small.

Figure 14: Comparison of Boston and New Yorkalgorithms

The Boston algorithm performs better thanthe NYC algorithm in this scenario when num-ber of choices is larger than 12. This is higherthan it was in scenario one (11) and two (7).The increase in probability is relatively con-stant when number of choices is small andbecomes increasingly small when number ofchoices is closer to 21.

4.3 Comparison of Scenarios

Figure 15: Performance of New York algorithmin each scenario

As mentioned earlier, we expected the utili-ties in scenario one to be smaller than those inscenario two and larger than those in scenariothree. Interestingly, we see that the opposite istrue for small number of choices although thedifference between the scenarios is small. Atthe other end of the spectrum, we see the or-dering of the scenarios that was expected andthat the difference between them is larger.

9

Figure 16: Performance of Boston algorithm ineach scenario

For the Boston algorithm the we also ex-pected scenario one to do worse than scenariotwo and better than scenario three. In this casethe results are as expected across all numbersof choices and the differences between them isabout the same as in the New York algorithmwhen number of choices is close to 21.

Figure 17: Comparison of Boston and New Yorkalgorithms in each scenario

We see that the probability that a randomutility taken from the Boston distribution islarger than one taken from the New York dis-tribution increases as the number of choices in-creases in all three scenarios. When the numberof choices is small the Boston algorithm per-forms best in scenario two and worst in scenariothree. This difference between the three scenar-ios becomes much smaller when the number ofchoices is larger.

5 Future Work

The methodology with which we create theacceptance rates for the Boston algorithm re-quires additional study. In practice we normallysee acceptance rates for schools remain largelythe same from year to year with sometimes asmall downward trend. However currently inthe simulation the acceptance rates for schoolscan vary tremendously from year to year. Theprogram in the future should be modified to findan equilibrium acceptance rate for each schoolthat only varies a small amount over time.

It would also be beneficial to the generalstudy of the algorithms to create additional dis-tricts based on other cities such as New Yorkor Chicago to see how performance of the algo-rithms varies in these cities compared to Wash-ington D.C. We would also like to study howcharacteristics of the school district such as den-sity, demographics, average commute times andaverage school qualities affects how well the al-gorithms are able to match students to theirschools of choice. Currently with just the databased on Washington we are only able to testout some of these effects by altering how stu-dents weigh their preference for academic qual-ity vs. distance to schools.

Another thing to study would be to lookat other ways in which students’ priority num-bers at schools are determined. In this paperfor all simulations we had students’ prioritiesat schools randomly determined as in a lottery.However, many school systems do not generatepriorities completely randomly and instead alsohave schools creating preferenes for certain stu-dents. One example of this is allowing studentsto be guaranteed their “home school,” whichis generally the school they are located clos-est too. One interesting alteration to the algo-rithms would be to add in a check so that if astudent lists their neighborhood school as theirtop choice, their admittance would be guaran-teed.

We would also like to examine the per-formance of a third method of matching stu-dents to schools. This third method would begiven by running a linear programming problemsolver that is set up to maximize student utility.It would be interesting to see if it does beat, andif so by, how much, the New York and Boston

10

algorithm distributions in terms of maximizingstudent utilities. We would also look at howmany students are receiving their top choices.Running a linear program in real life may provemore difficult, as it would be harder to get stu-dents to express how much they like each schoolwith scores as they do in this simulation. Whilethe Boston and New York algorithms are runusing only ordinal preferences, additional infor-mation is required in this case.

References

[1] Abdulkadirolu, Atila, Parag A. Pathak, andAlvin E. Roth, “The New York City HighSchool Match, American Economic Review,Papers and Proceedings, 95,2, May, 2005,364-367

[2] Abdulkadirolu, Atila, Parag A. Pathak,Alvin E. Roth, and Tayfun Snmez, “TheBoston Public School Match, AmericanEconomic Review, Papers and Proceedings,95,2, May, 2005, 368- 371

[3] Abdulkadirolu, Atila, Parag A. Pathak, andAlvin E. Roth, “Strategy-proofness versusEfficiency in Matching with Indifferences:Redesigning the NYC High School Match,American Economic Review, 99, 5 (Decem-ber), 2009, 1954-1978.

[4] Hoxby, Caroline M. “School Choice andSchool Productivity (Or, Could SchoolChoice be a Rising Tide that Lifts AllBoats?,” in C. Hoxby, ed. The Economicsof School Choice, Chicago: University ofChicago Press, 2003

[5] Hoxby, Caroline M. “School Choice andSchool Competition: Evidence from theUnited States, Swedish Economic PolicyReview, 10.2, 2004

[6] Roth, A.E. “The National Resident Match-ing Program as a labor market,” JAMA.Journal of the American Medical Associa-tion, April 3, 1996, 275, No. 13, (Pulse),1054-1056

APPENDIX

Table 1: Performance of both algorithms after oneround (scenario one)

11

Table 2: Performance of both algorithms after allstudents matched (scenario one)

Table 3: Probability that a random utility afterone round is greater from the Boston al-gorithm distribution than the New Yorkalgorithm distribution (scenario one)

Table 4: Probability that a random final utility isgreater from the Boston algorithm distri-bution than the New York algorithm dis-tribution (scenario one)

Table 5: Performance of both algorithms after allstudents matched (scenario two)

12

Table 6: Probability that a random utility isgreater from the Boston algorithm distri-bution than the New York algorithm dis-tribution (scenario two)

Table 7: Performance of both algorithms after allstudents matched (scenario three)

Table 8: Probability that a random utility isgreater from the Boston algorithm distri-bution than the New York algorithm dis-tribution (scenario three)

13