Why Do College Coaching Interventions Work? Scott Carrell, UC Davis and NBER Bruce Sacerdote, Dartmouth College and NBER 1

Why Do College Coaching Interventions Work?

Scott Carrell, UC Davis and NBER

Bruce Sacerdote, Dartmouth College and NBER

1

2

US College Completion Ranks 11-13thth In OECD

00.2

0.4

0.6

1970 1980 1990 2000Year

US NZ UK SWIITA FIN SWE

Cascio Gordon Clark JEP 2008

3

US Dept of Ed Trio Programs

$879 million in FY 2011 GEAR up, Upward Bound, Talent Search

Fundamental tenets: Catch students early..by 8th grade to get them college ready Avoid discussion of randomized evaluation Bruce and Scott don’t know what they are talking about

4

US Economist/Ed School Programs

Send letters to top students (Hoxby and Turner) Auto fill the FAFSA (Bettinger Long Oreopolous

Sabonmatsu) Send text messages (FAFSA reminders or deadline

reminders) (Castleman and Page) Offer remote coaching or application help (Phillips and

Reber) Carrell and Sacerdote: in person help, financial

incentives, “likely letters” Clearly different populations in all of these

5

2010 Cohort: Frequency Counts 10th Grade Math Scores for Non College Goers

05

001

000

150

02

000

250

0F

req

uenc

y

1100 1120 1140 1160math_score

05

00

10

00

15

00

20

00

25

00

Frequency

1100 1120 1140 1160 1180math_score

No College College

6

The Mission

Ask whether simple interventions (bonuses, mentoring, likely letters) very late in the game can have a meaningful impact on long term outcomes Attending, persisting, graduating, income Use Clearinghouse data to track into College

Are these marginal college going students very different than the average?

For whom does it work? Try to infer mechanisms using baseline interests, preparation, personality measures SAT Questionnaire, survey data

7

The Answer

Yes If you are female Or a guy who has not taken the SATs If you do not have strong parental and teacher support for

completing applications If you are not extraverted

What doesn’t work Likely letters Letters of encouragement (Delaware, NH) Cash Bonuses alone

8

Hypotheses

Behavioral Fear of process Lack of easily obtained information Procrastination/disorganization

Largely rational: have already optimized

Need expert/ adult help

9

What is the Intervention

Guidance counselors identify high school seniors at risk of not applying to college Ideally students who have expressed interest in applying But have taken few steps Typically identify in mid-December

We randomly choose half for the treatment group Guidance staff then invites treatment group to participate

10

What is the Intervention (2) We visit once per week for 3-5 weeks

– Pair up each HS student with a Dartmouth student– 1.5-4 hours per session – Start and complete applications– Send transcripts as needed– Pay application fees– Complete common apps, two year schools– Start the FAFSA– Sign up for SATs & send scores

Incentives– Free coaching– App fees paid – $100 cash bonus upon completion– Get out of class– Pizza

The Process

11

Mission Accomplished (?)

12

13

Challenges Faced By HS Students And Guidance Counselors

Horrible internet access and computers– In first year, many schools had BLOCKED the application

sites for all NH Community Colleges– When that was fixed, we still had credit card problems– In some cases we brought our own laptops and MiFi cards

Students and parents unfamiliar with Common App, FAFSA, online forms

Students were often selected to be the procrastinators

14

Treatment and Control Standardized Reading Scores

0.1

.2.3

.4.5

kden

sity

re

adin

gST

D

-4 -2 0 2x

Treatment Control

15

Standardized Math Scores: Treatment Group Versus All Non Experimental

0.1

.2.3

.4.5

kden

sity

ma

thS

TD

-4 -2 0 2 4 6x

Treatment All Other

Control Group

Mentoring Treatment

Variable Obs Mean Std. Dev Obs MeanStd. Dev

Accepted Treatment 902 0 0 871 0.454 0.49810th Grade Math Score (Standardized) 798 -0.480 0.937 778 -0.286 0.94310th Grade Reading Score (Standardized) 799 -0.436 0.928 772 -0.278 0.966Math > 50th Percentile in State 798 0.312 0.464 778 0.335 0.472Reading > 50th Percentile in State 799 0.350 0.477 772 0.398 0.490Math >75th Percentile 798 0.164 0.371 778 0.185 0.389Reading > 75th Percentile 799 0.213 0.410 772 0.224 0.417Free and Reduced Lunch Eligible 902 0.277 0.448 871 0.286 0.455Male 902 0.548 0.498 870 0.575 0.495Non-white 902 0.173 0.378 871 0.201 0.401Any College (Clearinghouse) 902 0.438 0.496 871 0.592 0.492Four Year College (Clearinghouse) 902 0.169 0.375 871 0.276 0.447Persist for First Two Years Post Grad 902 0.195 0.397 871 0.240 0.427Persist in a Four Year College 902 0.094 0.292 871 0.115 0.319Enrolled 3+ Semesters

902

0.237

0.426

871

0.292

0.455 Students are randomly assigned to treatment within high school. Data include 2009-2014 cohorts.

Table 1: Summary Statistics for Treatment and Control Groups

17

Table 2: Show Treatment Status Unrelated to Pre-Treatment Characteristics

Students are randomly assigned to treatment within high school. Data include 2009-2014 cohorts. Regressions include high school*cohort dummies which is the level at which randomization occurred. Standard errors are clustered at the high school*cohort level. Regressions include birthyear*cohort dummies to control for students' age within grade.

(1) (2)

Treatment Status Men

Treatment Status Women

Standardized 10th Grade Math Score 0.001 0.041

(0.012) (0.025)

Standardized 10th Grade Reading Score -0.025+ -0.006

(0.014) (0.020)

Free Reduced Lunch Eligible -0.043 0.073

(0.027) (0.046)

Student is Nonwhite 0.019 -0.038

(0.032) (0.057)

Observations 1216 866R-squared 0.355 0.321F Pre-Treat Variables 1.281 2.109

p-value 0.294 0.098

(1) (2) (3) (4) Apply to

Any College

(Survey)(OLS)

Apply to Any

College (Survey)

IV Estimate

Women Apply to

Any College

(Survey) OLS

Men Apply to Any

College (Survey)

OLS

Mentoring 0.274** 0.375** 0.294** 0.243**Treatment

(0.050) (0.068) (0.049) (0.078)

Observations 859 859 391 468R-squared 0.234 0.309 0.293 0.231

18

Effect of Treatment Status on Applying

Survey data with 50% response rate. Control mean is .72.Standard errors in parentheses, includes high school* cohort f.e. , age controls, male+ significant at 10%; * significant at 5%; ** significant at 1%

(1) (2) (3) (4) (5) Whole

SampleWomen Men Did Not

Take SAT

Took SAT

Effects on Enrollment Any College

Mentoring Treatment (OLS) 0.060** 0.146** 0.007 0.083** 0.035

(0.018) (0.042) (0.025) (0.026) (0.035)

Transcript Only (OLS) -0.005 0.005 0.000 0.035 -0.049 (0.019) (0.034) (0.021) (0.034) (0.035) Mentoring Treatment (IV) 0.133** 0.299** 0.017 0.160** 0.086

(0.041) (0.087) (0.061) (0.047) (0.085)

First Stage for IV Mentoring Treatment 0.463** 0.500** 0.429** 0.511** 0.444** (0.039) (0.044) (0.042) (0.033) (0.070)Observations 2,623 1,114 1,509 1,453 1,170

19

Baseline Treatment Effects on Enrollment in Any College

Outcome variable is a dummy equal to 1 if the student has any enrollment in college including 2 year or four year colleges. Outcome variables are based on the Nation Student Clearinghouse data. Data include 2009-2014cohorts. Regressions include high school*cohort dummies which is the level at which randomization occurred. Standard errors are clustered at the high school*cohort level. ** p<0.01, * p<0.05, + p<0.1

Effects on Enrollment Four Year College

Whole Sample

Women Men Did Not Take SAT

Took SAT

Mentoring Treatment (OLS) 0.057** 0.107** 0.020 0.103** -0.005 (0.018) (0.031) (0.028) (0.026) (0.033)Transcript Only (OLS) 0.001 0.007 0.003 0.002 -0.038 (0.015) (0.022) (0.028) (0.012) (0.030)Mentoring Treatment (IV) 0.125** 0.222** 0.047 0.202** -0.018

(0.037) (0.062) (0.068) (0.048) (0.083)

First Stage for IV Mentoring Treatment 0.463** 0.500** 0.429** 0.511** 0.444** (0.039) (0.044) (0.042) (0.033) (0.070)Observations 2,623 1,114 1,509 1,453 1,170

20

Baseline Treatment Effects on Enrollment in A Four Year College

(1) (2) (3) (4) (5) Women Women Women Men Women No SAT Data Enrolled in

3+ SemestersEnrolled Any College Both School Years

Post Graduation

Enrolled Four Year

College Both School Years

Post Graduation

Enrolled Four Year

College Both School Years

Post Graduation

Enrolled Second Year Conditional on Enrolled First Year

Mentoring Treatment

0.129*(0.053)

0.105*(0.042)

0.097**(0.030)

0.014(0.041)

-0.040(0.066)

Observations 535 535 535 445 263R-squared 0.172 0.123 0.105 0.220 0.165

21

Treatment Effects on Persistence in College (Women)

Outcome variables are four different ways to measure persistence into the second year of college. Sample is limited to women in the 2009-2012 cohorts.

(1) (2) (3) (4) (5) (6)

SAT Questionnaire Measure

Coefficients on Treatment *SAT/

Survey Measure

Coeff on Treatment

Indicator

Coeff on SAT/

Measure

N Mean SAT/Surv indicator regressed

on Male Dummy

Do Not Need Help With Educ Planning

-0.116*(0.059)

0.126**(0.058)

0.049(0.039)

1302 0.829 0.015***(0.004)

Survey Measure

Parents Help With -0.131* 0.118** 0.133*** 724 0.468 0.014College Applications (0.067) (0.045) (0.041) (0.037)

Teacher Helps With -0.165* 0.112*** 0.089 646 0.172 -0.023

College Applications (0.091) (0.030) (0.062) (0.030) Guidance Counselor -0.009 0.0541 0.037 724 0.312 -0.0982**

Helps with College Application

(0.069) (0.037) (0.057) (0.034)

22

Evidence on MechanismsInteraction of Mentoring Treatment with Sources of Assistance on Applications

Dependent Variable is Enrollment in Any College .

(1) (2) (3) (4) (5) (6)Survey Measure Coeff on

Treat*Survey

Measure

Coeff on Treat

Indicator

Coeff on Survey

Measure

N Mean Survey indicator regressed

on Male Dummy

Log (Hourly wage at Age -0.159* 0.545** -0.052 354 2.931 0.307***30 w. only HS Diploma) (0.078) (0.233) (0.099) (0.052) Log (Tuition+Fees -0.019 0.246 -0.023 506 9.052 0.032Community College) (0.036) (0.310) (0.031) (0.091) Log (Tuition Fees NH -0.028 0.343 0.015 502 10.076 -0.040Public University) (0.030) (0.295) (0.023) (0.085) Need College Degree for -0.010 0.027 0.273*** 663 0.777 -0.089***Stated Career Goal (0.112) (0.094) (0.055) (0.032)

23

Interaction of Mentoring Treatment with Beliefs About Wages/ Tuition


24



(1) (2) (3) (4) (5) (6)

Survey Measure Coefficients on

Treatment*Survey Measure

Coefficient on

Treatment Indicator

Coefficient on

Survey Measure

N Mean Survey indicator regressed

on Male Dummy

Individual Measures

Likes to meet new people -0.305*** 0.280*** 0.150** 530 0.723 -0.096** (0.086) (0.085) (0.055) (0.039)Enjoy Amusement Rides -0.287** 0.259** 0.097 530 0.696 0.031 (0.136) (0.103) (0.087) (0.040)

Composite Measures Meets Deadlines/ Organized

0.083(0.189)

0.030(0.082)

0.096(0.133)

530 0.343 0.011(0.022)

Adventuresome -0.275 0.239 0.144 530 0.657 0.017 (0.179) (0.143) (0.146) (0.021)Self-Esteem -0.097 0.136 0.143 552 0.672 0.007

(0.128) (0.096) (0.092) (0.028)

25

Interaction of Mentoring Treatment with Personality Traits


26

Interaction of Mentoring Treatment with Personality Traits


27

Some Takeaways We find a boots on the ground approach is effective

– Appears to compensate for lack of or non take up parental or teacher help

– No evidence that its more effective for people who struggle with deadlines, planning, organization

– Helps students less far along in process, ie non SAT takers– Much more robust effects for the women..lack of effects for men very

much related to labor market opportunities

Can’t find evidence for effects from…– Likely letters/ letters of encouragement– Cash bonuses along..Though cash bonuses may be very important for

takeup– Texting of CCSNH students has not yielded effects– Nor did letter of encouragement to DE graduated seniors

28

More Takeaways We find evidence in favor of optimization story and

need skilled help story

We don’t find evidence for super naïve behavioral story Or that failure to attend is driven by forgetfulness or lack of

organization But parents/mentors could indeed be solving this

We suspect that letter based interventions or simple text reminders may only work with specific groups or in specific contexts

29

Also Mention If Time Huge drop off in participation when we removed the

cash bonus

Men very likely to say they have a job they prefer to college And no evidence in ACS of OLS return to two years of college

Still have low average persistence in both treat and control

Can’t make the Big Five measures work

30

A Very Few Words on Returns to CollegeOreopoulos and Salvanes

31

Caveats on Earnings Returns to College Large number of papers but sources of ID very

different– We haven’t actually shown that the marginal students in

current college going interventions have earnings gains– Bettinger, Gurantz, Kawano and I having trouble finding

earnings effects from winning the Cal-Grant– The one thing we have learned is just how ridiculously noisy

the outcome is

32



33

Do We Affect Type of College Attended? Treatment 0 1 Nashua Community College 25 24 Manchester Community College 15 22 Nhti - Concord's Community College 14 16 Great Bay Community College 11 8 Plymouth State University 2 8 Rivier College 4 7 University Of New Hampshire 7 6 Southern New Hampshire 0 5 Hesser College - Manchester 4 4 Community College Of Vermont 1 3 Colby Sawyer College 1 2 Hesser College -Nashua 3 2 Northern Essex Community College 0 2 University Of Massachusetts At Lowell 0 2 Arkansas State University - Newport 0 1 Pennsylvania State University 0 1 Total Shown 87 113 Total in College 114 130

34

Do We Affect Type of College Attended? (2)

(1) (2) (3) Institution Attended

is Masters Degree Granting: Both

Genders At Any College

Institution Attended is

Masters Degree Granting: Women

At Any College

Acceptance Rate of College

Attended: Both

Genders Treatment 0.002 0.038 -2.237

(0.040) (0.081) (2.811)

Observations 383 171 145 R-squared 0.133 0.231 0.106

Outcome variables measured in IPEDS data. Sample only includes students in college and for whom we have IPEDS data

35

Split Sample By Test Score

(1) (2) (3) (4) Enrollment

Two Year College Women

Enrollment Four Year

College Women

Enrollment Two Year

College Men

Enrollment Four Year

College Men

Mentoring Treatment 0.137+

(0.067) 0.064

(0.042) 0.014

(0.057) 0.060

(0.041)

Reading Score > 50th Percentile in Treatment Group

-0.208* (0.090)

0.116* (0.052)

-0.054 (0.073)

-0.025 (0.047)

Transcript Only Group -0.044

(0.045) 0.003

(0.032) 0.024

(0.037) -0.026

(0.023)

Observations 788 788 1,080 1,080 R-squared 0.124 0.174 0.075 0.231

(1) (2) (3) (4) VARIABLES Women:

Enrolled in Any College

Men: Enrolled in

Any Year College

Women: Enrolled in Four Year

College

Men: Enrolled in Four Year

College Treatment 0.150 0.027 0.202 0.085

(0.074) (0.018) (0.091) (0.032)

Immigrant/ Refugee -0.039 0.057 -0.211 0.099

(0.384) (0.419) (0.077) (0.226)

Immigrant*Treatment 0.002 0.089 -0.071 -0.260 (0.285) (0.219) (0.066) (0.173) Observations 136 144 136 144 R-squared 0.235 0.182 0.134 0.118

36

Interaction of Treatment with Immigration Status

Data are from Manchester West 2010,2011 Cohorts. Sample is roughly 9% immigrants.

37

Mechanisms Bonus? Incentive?

Same or Cross Gender Mentoring?

What do participants say?

More effective in schools with fewer resources?

38

Evidence From 2012 Cohort (Coaching Plus $100 Bonus Versus Bonus Alone)

39

Post-Survey Evidence on $100 Bonus

$100 bonus affect Decision to Complete Program?

Freq.

Aware, no effect

11 Initially motivating not long run factor

2

Not expecting

4 Important

2

Total 19

Take Up Rates Within Mentoring Group

41

What Aspects Helped the Most? 19 of 19 mentioned in person help with applications

5 of 19 mentioned the $100 bonus– 3 of 7 men versus 2 of 10 women

12 of 19 mentioned our paying for application fees– Rational bc a lot more than $100 for some students

Treatment Effect by High School

43

Average College Going Versus Effect Size For Women

44

Average College Going Versus Guidance Counselors Per Student

Manchester WestManchester WestDover

Portsmouth

Pinkerton

Manchester WestManchester WestManchester West

Nashua NorthNashua North

KearsargeKearsarge

LebanonLebanon

Pinkerton

Manchester WestLebanon

Kearsarge

Nashua South

Nashua North

KearsargeKearsarge

Nashua North

Nashua South

Manchester WestManchester West


Pinkerton

Portsmouth

Lebanon

Nashua North

Manchester West

Nashua North

Manchester West

Nashua North


Nashua South

Manchester WestDover


Kearsarge

Nashua South

Portsmouth

Nashua North

Dover

Lebanon

Portsmouth

Nashua North

Dover

Nashua South

LebanonManchester West

Nashua South

Nashua North

Dover

Nashua North

Kearsarge


PinkertonPinkerton

Londonderry

Nashua North

Nashua South

Manchester WestLondonderry


Manchester West

Nashua North

Portsmouth

Manchester West

Nashua North


Portsmouth

Lebanon

Kearsarge

Manchester West

Kearsarge

Nashua North


Nashua North

Lebanon

Nashua North

Manchester West

Pinkerton



Pinkerton






Lebanon

Nashua North

Nashua South


Kearsarge

Dover

Portsmouth

Manchester West


DoverDoverManchester West

Kearsarge

Portsmouth


Pinkerton

Nashua North

Manchester West

Nashua North


Pinkerton

Nashua North

Portsmouth

Manchester West

Kearsarge

Portsmouth

Kearsarge

Manchester West

Nashua North

Manchester WestManchester WestLebanon

Pinkerton

Portsmouth

Manchester WestLondonderry

Pinkerton

Manchester West

Pinkerton

Nashua South

Nashua North

DoverDoverPinkerton

Nashua South

Nashua North

Pinkerton

Nashua South


Nashua North

Portsmouth

Nashua NorthNashua NorthNashua North


Nashua North

Dover

Nashua North

Kearsarge

Manchester West

Nashua SouthNashua South

Manchester West

Nashua South

Dover



Nashua North

Nashua South

Nashua North

Nashua South

DoverManchester West

DoverManchester WestManchester WestManchester West

Nashua South

Dover

Lebanon


Manchester West

Pinkerton



Nashua North

Manchester West

Nashua North

Manchester West

Nashua South

Manchester West

PinkertonPinkerton

Nashua South

PinkertonDover

Nashua North

Nashua South

Portsmouth


Lebanon

Nashua South


Pinkerton

Nashua South


Londonderry


Nashua South

Dover



DoverPinkerton

Portsmouth

Manchester West


Pinkerton



Nashua North


Nashua South

Portsmouth

Manchester West

Nashua South


Portsmouth

Nashua South

Nashua North

Nashua South


Nashua SouthNashua SouthNashua SouthNashua South

Manchester West

Nashua North

Kearsarge


Pinkerton


Nashua South

Nashua North

Nashua South

Manchester West

Nashua North

Nashua South

Manchester West

Pinkerton

Manchester West

Pinkerton

Nashua North

Manchester West

PinkertonPinkerton

Lebanon

Nashua North

Lebanon


Nashua North

Manchester West

PinkertonDover

Nashua South

Lebanon

Pinkerton


Kearsarge


Pinkerton

Manchester West

Nashua North

Nashua South

Nashua North

Nashua South

Pinkerton

Manchester West

Pinkerton

Portsmouth

Kearsarge

Nashua South

Manchester West


Nashua North

Manchester West


Nashua NorthNashua NorthNashua NorthNashua North

Manchester West

Nashua South

Nashua North

Lebanon

Pinkerton

Londonderry

Kearsarge

Manchester West

Kearsarge

Nashua South

Kearsarge



PinkertonPinkertonPinkerton

Nashua South

Kearsarge

Manchester West

Nashua South


Nashua South

Portsmouth

Nashua South


Manchester West

Pinkerton


Nashua South

Lebanon

Kearsarge

Manchester West

Nashua North

Lebanon

Pinkerton

Nashua South

Manchester West

Nashua South

Kearsarge

Manchester West

Nashua North

Kearsarge


Nashua North


DoverPinkerton


Pinkerton

LondonderryManchester West

Nashua North

PinkertonPinkerton

Nashua South

Pinkerton

Nashua South

Nashua NorthNashua NorthNashua NorthNashua North

Pinkerton

Nashua South

Pinkerton

Nashua NorthNashua NorthNashua NorthNashua NorthNashua North

Nashua South

Nashua North

Manchester West

Nashua South

Nashua North


Kearsarge

Pinkerton

Portsmouth


Nashua South

Manchester West

Pinkerton

Nashua South

Manchester West

PinkertonPinkerton

Nashua South

Manchester WestManchester WestManchester WestManchester WestManchester West

Nashua North

Manchester West

Nashua South

Nashua North

Pinkerton

Manchester West

Nashua South

Manchester West

Nashua South

Nashua North


Nashua North

Manchester West

Kearsarge

Pinkerton

Nashua South

Manchester West

Pinkerton


Nashua South

Pinkerton


Portsmouth

Manchester West

Nashua South

Manchester West

Nashua North

Dover


Pinkerton


Portsmouth

Manchester West

Nashua South

DoverPinkerton

Manchester West

Nashua South


Pinkerton

Nashua South

Dover

Nashua North

Londonderry

Nashua South

Nashua North

Kearsarge

Manchester WestManchester WestManchester WestManchester West

Pinkerton

Nashua South

Nashua North

Manchester West

Pinkerton


Pinkerton

Manchester West

Nashua South

Manchester West

Nashua South

Portsmouth

Manchester West

PortsmouthPortsmouth

LondonderryLebanon

Dover

Nashua North

DoverPinkerton


Dover

LebanonLebanon


Pinkerton

Nashua North

Pinkerton


PinkertonDover

Kearsarge

Nashua South


DoverPinkertonPinkerton

Nashua South

Pinkerton

Lebanon

Pinkerton




Portsmouth

KearsargeKearsarge

Nashua South

Manchester West

Nashua South

Pinkerton

Kearsarge

Nashua South



PinkertonPinkertonDover

Nashua South

LondonderryManchester WestManchester West

Nashua North

Portsmouth


Lebanon

DoverDoverDover

Nashua South

Londonderry

Nashua South

Pinkerton

Manchester West


Pinkerton

Nashua SouthNashua SouthNashua South


Manchester WestManchester WestManchester WestManchester WestManchester WestManchester West

Pinkerton

Portsmouth


Nashua South

Pinkerton


Manchester West


Lebanon

Nashua South

PinkertonDoverDover

Manchester West

Nashua South

Pinkerton

Nashua North

Kearsarge


Nashua South



PinkertonPinkertonPinkertonDover

Nashua South

Manchester West

Pinkerton

Kearsarge

Dover

Nashua South

Dover

Nashua South

Portsmouth




Nashua South

Portsmouth


Portsmouth

Nashua North

Portsmouth

Pinkerton

Nashua South

Manchester West

Nashua North

Portsmouth

Dover

Nashua South

Manchester West

Nashua South

Pinkerton

Nashua South



Manchester West

Nashua South


Pinkerton

Nashua South

Dover

Nashua SouthNashua SouthNashua SouthNashua SouthNashua South

Nashua North

DoverPinkerton

Manchester West

Nashua North

Portsmouth

LondonderryDover

Manchester WestManchester WestManchester WestManchester WestManchester West

Nashua South

Nashua North


Lebanon


Lebanon

Pinkerton

Nashua South


Nashua North

Nashua South

Portsmouth

Nashua South


Manchester West


Londonderry

Nashua South

Pinkerton


Portsmouth

Nashua South

Portsmouth

PinkertonPinkertonPinkertonPinkerton


Manchester WestLebanonLebanon

Nashua South

Manchester West



Nashua South

PinkertonPinkerton


Manchester West

Kearsarge

Nashua South


Manchester West

Nashua South

Manchester West

Portsmouth

Nashua South

Nashua North

Manchester West


Manchester West

Nashua South



LebanonLondonderry

Dover

Lebanon


Pinkerton

Nashua South


Pinkerton

Nashua North


Nashua South

Portsmouth


Nashua North


Nashua North


Pinkerton

Manchester West

Nashua South


Nashua South

Pinkerton


Portsmouth

Nashua North

Manchester West

Pinkerton

Nashua South

Pinkerton

Nashua South

Portsmouth

Manchester WestManchester WestManchester WestLebanon

Nashua North

Dover

Nashua North

PinkertonPinkerton


Portsmouth



DoverPinkerton

Nashua North

Manchester West


LebanonManchester WestManchester West

Nashua South

Manchester WestManchester WestDover

Pinkerton

Londonderry

Pinkerton

Portsmouth





Nashua North

Manchester West

Nashua North

Nashua South

Nashua North

Nashua South

Manchester West

Nashua South


Nashua South

Nashua North

Nashua South

Manchester West

Nashua South

Pinkerton

Nashua South

Portsmouth

Manchester WestLondonderryManchester WestDover

Nashua South

Pinkerton


Nashua South


Nashua North



Dover


Nashua North

Manchester West

Pinkerton


Nashua North

Manchester West

Nashua South


Nashua North

Nashua South


Nashua North


Manchester West

Nashua South

Kearsarge

Manchester West

Nashua South

Manchester West

Pinkerton

Nashua South

Nashua North



Nashua South

Manchester West

Nashua South


Manchester West

Nashua South


Pinkerton

Nashua South

Nashua North

Nashua South

PinkertonDover


Portsmouth


Manchester West

Nashua South





Manchester West

Nashua North

Pinkerton

Nashua SouthNashua SouthNashua SouthNashua South

Manchester West


Nashua North

Nashua South

Nashua North

Pinkerton

Nashua North

Nashua South


Nashua South

Portsmouth

PinkertonPinkerton


Pinkerton


Nashua South

DoverPinkerton


Pinkerton

Nashua South

Londonderry

Nashua South

Nashua North

Pinkerton


Pinkerton

Manchester West

Nashua South

Nashua North

Pinkerton

Manchester West


Pinkerton

Nashua South

Dover

Lebanon

Pinkerton

Manchester West

Pinkerton


Nashua North

Pinkerton


Manchester West

Pinkerton

Portsmouth

Pinkerton

Manchester WestManchester WestLebanon

Nashua South


PinkertonPinkerton

Portsmouth


PinkertonDover

Nashua North

Nashua South

Nashua North

Manchester West


Pinkerton

Manchester West

Nashua North


Manchester West

Nashua South


Pinkerton


Pinkerton



Pinkerton

Manchester West

PinkertonPinkerton

Nashua South

Nashua North

Kearsarge

Manchester West

Nashua North

Nashua South



Manchester West

Nashua South

Dover

Nashua North

Manchester West

Pinkerton

Nashua South

Portsmouth

Nashua South

Kearsarge


Nashua South

Nashua North

Nashua South

Portsmouth

Nashua North

DoverDoverManchester WestManchester WestManchester West


Pinkerton


Pinkerton


Pinkerton

Kearsarge

Lebanon

Pinkerton

Nashua South

DoverDover

Nashua South

PinkertonDover

Nashua South

Pinkerton

Nashua South

Dover

Nashua South


Nashua North


PinkertonDoverDover

Pinkerton

0.1

.2.3

.4e

stim

ate

d_e

ffect

.003 .004 .005 .006guidance_per_student

Mentoring Treatment Interaction with Sources of Disadvantage



48

Stories from the HS Students > hey Cambell, its Daniel M from west, one of the

student you hepled apply for college. If i remember correctly, you told me to let you know what college i get into. Well, i was recently accepted to NHTI in Concord.

There are still some things i need to take care of, but its nothing serious.

> P.S. Thanks for you help. Sorry for all the flirting

49

Cross Tab of Student Male and Assigned Male Mentor

assigned_male_mentor

Student is

Total Male 0 1

0 76 73 149 1 61 95 156

Total 137 168 305

Mentors were assigned on a first come first served basis, but when multiple arrivals occurred at the same time, we had a modest bias towards same gender pairings. Regressions include a dummy for being assigned to treatment but not showing up to be assigned a mentor.

Is Cross Gender Mentoring More Effective?

51

Four Theories As To Why It Only Works for Women

Treatment is actually a complement to ability or persistence– Look for other ways in which treatment might interact with sources

of advantage, help or test scores

Guys are primed to either be near the top of an endeavor or quit– Don’t respond well to feedback that they are in the middle

Labor economics – Short run returns to college in NH are lower for guys

Are guys negatively selected into the treatment group? – In part bc of $100 bonus?

52

Survey Data on Male And Female Study Subjects

Women Men N Mean SD N Mean SD No Sat Data

1077 0.57 0.49 1463 0.57 0.50

Sat Math

95 42.77 8.33 133 45.42 9.38

Sat Verbal

95 44.02 8.48 133 45.14 10.47

Role Model College

43 0.16 0.37 61 0.21 0.41

Self Worth High

45 0.51 0.51 67 0.60 0.49

Remember Dates

44 0.30 0.46 67 0.31 0.47

Forget Deadlines

44 0.32 0.47 67 0.30 0.46

Have Control In Life 45 0.56 0.50 67 0.54 0.50

53

Survey Data on Male And Female Study Subjects (Aspirations and Expectations)

Women Men N Mean SD N Mean SD Aspire College4 46 0.80 0.40

68 0.81 0.40

Role Model College 43 0.16 0.37

61 0.21 0.41

Sat Aspire College4 95 0.65 0.48

133 0.65 0.48

Teachers Expect College 43 0.81 0.39

66 0.71 0.46

Parents Want College4 43 0.67 0.47

67 0.52 0.50

Talked Future Plans Teacher 46 0.46 0.50

68 0.43 0.50

Talked Future Plans Parent 46 0.85 0.36

68 0.85 0.36

(1) (2) (3) (4) Log Total

Income Men NH

Log Total Income

Women NH

Log Total Income Men

All Other States

Log Total Income

Women All Other States

High School 0.399** 0.409** 0.429** 0.477**

(0.055) (0.103) (0.003) (0.005)

One to Three Years of 0.686** 0.667** 0.726** 0.761** College

(0.058) (0.104) (0.003) (0.005)

Four Plus Years of 1.081** 0.951** 1.251** 1.252** College

(0.055) (0.102) (0.003) (0.005)

Constant 9.955** 9.325** 9.748** 9.146** (0.051) (0.099) (0.003) (0.004) Observations 4,674 4,570 1,129,025 1,050,794 R-squared 0.140 0.049 0.168 0.106 F Test HS=Some College

77.69 33.52 15889 9056

p-value 0 7.53e-09 0 0

54

How Do Returns to College Differ for Men Versus Women in NH (Ages31-40)?

55

How Do Returns to College Differ for 21-30 year old Men Versus Women in NH?

(1) (2) (3) (4) Log Total

Income Men NH

Log Total Income

Women NH

Log Total Income

Men All Other States

Log Total Income

Women All Other States

High School 0.343** 0.403** 0.345** 0.484**

(0.075) (0.100) (0.004) (0.005)

One to Three Years 0.339** 0.593** 0.405** 0.673** of College

(0.078) (0.101) (0.004) (0.005)

Four Plus Years of 0.663** 0.848** 0.839** 1.193** College

(0.077) (0.099) (0.004) (0.005)

Observations 2925 2898 828,881 794,172 R-squared 0.033 0.046 0.055 0.095 F Test HS=Some College

0.00493 14.49 414.6 3331

p-value 0.944 0.000144 0 0

56

Other Interventions Find Effects Differ By Gender?

Moving to Opportunity only saw decreases in crime and increases in school engagement for girls

Angrist and Lavy bonuses in Israel for obtaining Bagrut

Study one of Oreopoulos and Angrist bonuses plus mentoring for U of Toronto students– Study two did not find any effects

Dynarksi Ichino find bigger effects of offereing finacial aid for women

Job Training programs affect women more

57

Other Interventions Find Effects Differ By Gender?

But not at all clear this is true for charter school studies– Boston area charter studies find bigger effects for boys

Are effects bigger for students who are weaker in that domain– Did boys already attend college up to zero marginal returns?– Girls need a shot of confidence or more help

Or is the treatment a complement to skill?– Is it ultimately about personality measures, follow through– Our intent is to collect ex-post survey measures of personality

traits, patients, follow through

58

Additional Interventions Send one group a letter offering admission to their

closest public college and also community college– We’ve had fantastic cooperation from admissions officers at

NH institutions (UNH, UNH Manchester, SNHU)– Not much evidence of an effect yet

Compare to Hoxby Turner which has large effects from letters sent to students in top 10% of SAT / ACT distribution– Our population is very different and perhaps needs much

more in person help hand-holding

59

Additional Interventions (2) Have students meet with Admissions Officers from

Community College System

Have Lets Get Ready run the program. Adds a serious SAT component Ask how much results change when program is scaled up/ PI is

no longer involved And when population of students is broadened

60

Additional Interventions (3) Use text messaging to Community College students to

prevent Summer Melt, encourage persistence, help students make the transition to 4 year institutions

61

Bottom Lines Big differences by gender..critical to better understand that..Quite

possibly its not a general boy phenomenom

These marginal students persist at same rate as average

Program is enormously successful for immigrants– And for resource poor high schools– Suggests that we are in some way compensating for missing component

rather than reinforcing advantage

Difficult or impossible to evaluate this sort of program using endogenous take up

May be real downsides to offering cash bonuses May be real upsides to “expensive” in person help

62

Could We Estimate the Effects for Women Without a Randomized Control Group?

(1) (2) (3) (4) Women

Four Year College:

Intended Treatment Versus All Nonexperi

mental

Women Any

College: Intended

Treatment Versus All Nonexperi

mental

Women Four Year

College: Intended


mental

Women Any

College: Intended


mental Assigned to Treatment Group -0.162** -0.004 (0.030) (0.034) Treated -0.138** -0.041

(0.040) (0.040)

Observations 21,042 21,044 20,934 20,936 R-squared 0.273 0.170 0.272 0.171

63

Compare to Existing Work HR Block (FAFSA) study

– The Role of Simplification and Information in College Decisions: Results from the H&R Block FAFSA Experiment• Eric P. Bettinger, Bridget Terry Long, Philip Oreopoulos, and Lisa Sanbonmatsu

– Moved enrollment rates from 26.8 to 34.5 percent– 7 percentage point effect in same age group– We have a more intensive intervention and (arguably/ debatably) twice the effect

size

COACH program (Avery and Kane)

Kane college mentoring in CA– 5 percentage point effect on enrollment rates

Berman Ortiz and Bos (ongoing in LA Unified) Phillips and Reber remote coaching in LA and CA

64

Estimate the Effects By Comparing Those That Accept Treatment to Those That Don't?

(1) (2) (3) (4) Women 4

Year College: Accepted

Treatment Versus Didn't

Women Any College:

Accepted Treatment

Versus Didn't

Men Four Year College:

Accepted Treatment

Versus Didn't

Men Four Year College:

Accepted Treatment

Versus Didn't Accepted Treatment 0.076 -0.035 0.070+ 0.017

(0.091) (0.092) (0.035) (0.067)

Constant 0.613* 0.228 -0.477** -0.601** (0.244) (0.466) (0.138) (0.040) Observations 259 259 341 341 R-squared 0.167 0.213 0.176 0.231

65

Modifications in Follow Up Work Try W/o the $100 bonus

– May actually increase treatment on treated for men– Eliminate noise of people signing up for wrong reason

Have Lets Get Ready run the intervention– Include SAT prep. Include juniors

Try letters of encouragement from college admissions departments

Extensive pre and post surveys to learn more about students personalities, career plans, stick-toitiveness

Try emails and texts to spur students to apply

66

Partnered with NH DOE NH DOE has been very supportive and flexible

– Commissioners and staff have leant their support to the project

Key pre-treatment and outcome variables collected by NH DOE– Nat’l Student Clearinghouse subscription– Test scores and demographics

Part of a larger Data Warehouse project underway

Future projects being designed jointly

What our High Schools Look Like

67

Documents

Why Do College Coaching Interventions Work? Scott Carrell, UC Davis and NBER Bruce Sacerdote, Dartmouth College and NBER 1