Program: Assessing Intended and Unintended … eap...2 4 6 8 10 12 14 16 18 Q1 Q2 Q3 Q4 d Pre-EAP Post-EAP Higher EAP participation is associated with higher levels of CST Advanced

Michal Kurlaender, University of California - Davis

Jacob Jackson, Public Policy Institute of California

California’s Early Assessment Program: Assessing Intended

and Unintended Consequences

1

Policy Analysis for California Education Seminar Series March 13, 2015

• Funding Agencies

– Institute of Education Sciences, U.S. Department of Education

– Bill and Melinda Gates Foundation

– Spencer Foundation

• State Agencies

– California State University Chancellor’s Office

– California Department of Education

– California Community College Chancellor’s Office

• Collaborators

– Jessica Howell, The College Board

– Eric Grodsky, University of Wisconsin

2

Acknowledgements

Presentation Outline

1. Background on the EAP

2. The Intended Impacts of the EAP

3. Unintended consequences and mechanisms of the EAP

4. Future Research on the EAP and College Readiness

3

BACKGROUND ON THE EAP

Section 1

4

High remediation and low graduation rates at CSUs

Percent of students requiring some remediation at CSU system and six-year graduation rates by cohort

Data from CSU Analytic Studies: http://www.asd.calstate.edu/performance/proficiency.shtml

Remediation Need

6 year grad rate

0%

10%

20%

30%

40%

50%

60%

70%

2000 2002 2004 2006 2008 2010

Pe

rc

en

t

First Time Freshman Cohort

5

Remediation need varies by race

Data from CSU Analytic Studies: http://www.asd.calstate.edu/performance/proficiency.shtml

Percent of students Requiring remediation at CSU in 2013

6

65%

55%

51%

49%

44%

37%

36%

21%

43%

0% 20% 40% 60% 80%

African American

Mexican American

Other Latino

Pacific Islander

American Indian

Filipino

Asian

White

Total

College Readiness by High School

0.2

.4.6

.81

Pro

port

ion R

ead

y in E

ng&

Ma

th

2.5 3 3.5 4 4.5HS GPA

All Schools Average

Readiness at Entry by GPA

7

“Readiness” Differs by High School

0.2

.4.6

.81

Pro

port

ion R

ead

y E

ng&

Math

2.5 3 3.5 4 4.5HS GPA

All Schools School A School B

Readiness at Entry by GPA

8

Early Assessment Program Overview

• Goals of EAP:

Provide an early signal to students about their college readiness

California State University collaboration with K-12

Provide 12th grade interventions

• Components of EAP:

11th grade testing (early assessment)

Professional development for teachers

Supplemental preparation for students

9

The EAP sends a college readiness signal to 11th grade students

• Optional 11th grade test in English and Mathematics

Optional 15 questions on 11th grade CST

Additional items developed by CSU faculty

Score based on CST augmented with EAP items

• Potential college readiness signals

Exempt from remedial courses at a CSU

Conditional Exempt from remedial courses at CSU

Not exempt from remedial courses at a CSU

10

Most students are not Ready for College Level Work

2013 EAP Test Results

11

55%

18%

18%

51%

21% 12%

7% 19%

0%

100%

Math English

Ready

Conditional

Not Ready

Did NotParticipate

THE INTENDED IMPACTS OF THE EAP

Section 2

12

Intended Consequences: Research Questions

• How does participation in California’s Early Assessment Program affect a student’s probability of requiring remedial coursework in college?

• Does the effect of the program differ for different students?

13




14

Data

• CSU Chancellor’s Office

– Six cohorts of first-time freshman applicants (2000/01 – 2005/06)

• California Department of Education

– Matched CSU applicants (enrollees) to CST scores, EAP participation, other student and high school characteristics

15

Creating a Counterfactual

Pre-EAP Cohorts Post-EAP Cohorts

Non-participants

Non-participants

Participants

16

Intent to Treat

Pre-EAP Cohorts Post-EAP Cohorts

Non-participants

Non-participants

Participants

17

Post-EAP Cohorts

Non-participants

Non-participants

Participants

Pre-EAP Cohorts

Non-participants

Impact of Participation

18

Pre-EAP Cohorts

Non-participants

Post-EAP Cohorts

Non-participants

Participants

Participants Non-Participants

90% Participation Schools

Impact of participation at high participation schools

19

The implementation of the EAP is related to lower likelihood of requiring remediation

20

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

English Math

Pr

ob

ab

ilit

y o

f N

ee

din

g

Re

me

dia

tio

n

Pre-EAP Post-EAP

Fitted Values for Remediation Rates (ITT)

-.018

-.020

Covariates, High School and College FE, 6 year sample

21

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

English Math

Pr

ob

ab

ilit

y o

f N

ee

din

g

Re

me

dia

tio

n

Pre-EAP Post-EAP

-.022

-.027

Fitted Values for Remediation Rates (For 90% Participation Schools)

Covariates, High School and College FE, 6 year sample

Participation in the EAP is related to lower likelihood of requiring remediation




22

Students with the lowest probability of needing remediation seem to benefit the most

-.0

4-.

02

0

.02

Est

imate

d E

ffe

ct

0 .2 .4 .6 .8 111th grade remediation need

Estimated EAP Effect at Different Levels of Need for English Remediation

23

Students with the lowest probability of needing remediation seem to benefit the most

Estimated EAP Effect at Different Levels of Need for Math Remediation

-.1

-.0

5

0

.05

Estim

ate

d E

ffe

ct

0 .2 .4 .6 .8 111th grade remediation need

24

Summary: Impacts of the EAP Program

25

• The EAP policy led to a modest reduction in the probability that CSU freshmen require remediation.

• EAP participation leads to a modest (1 to 2 percentage point) reduction in the probability that CSU freshmen require remediation.

• The program had larger effects for students who were less likely to need remediation

UNINTENDED CONSEQUENCES AND MECHANISMS OF THE EAP

Section 3

26

Unintended Consequences and Mechanisms: Research Questions

27

1. Does the EAP discourage CSU enrollment? 2. Does the EAP encourage math course-taking in 12th grade? 3. How does the EAP impact high school accountability measures? 4. Has there been an impact of the EAP on community colleges?


28


Most students are not Ready for College Level Work

2013 EAP Test Results

29

55%

18%

18%

51%

21% 12%

7% 19%

0%

100%

Math English

Ready

Conditional

Not Ready

Did NotParticipate

How did the EAP reduce the need for remediation?

Option 1: Students are motivated by a not-exempt signal to improve in their senior year.

Option 2: Students receiving a not-exempt signal are discouraged from attending a CSU.

30

Testing the impact of the college readiness signal

• Outcomes:

– CSU Application

– CSU Enrollment

• Predictor:

– Readiness signal (exempt status)

31

Example regression discontinuity effect

0.1

.2.3

.4.5

.6

P(A

pply

to C

SU

)

-60 -40 -20 0 20EAP English Score (0=Exempt)

32

Hypothetical Probability of Applying to a CSU by EAP English Score

Hypothetical 8 percentage point

difference

The English readiness signal has a small impact on CSU application

(only among math nonparticipants)

33

0.1

.2.3

.4.5

.6

P(A

pp

ly to

CS

U)


N= 671,861 Effect =.015 ** (.006)

Probability of Applying to a CSU by EAP English Score

The English readiness signal has no impact on CSU enrollment

34

0.1

.2.3

P(a

tte

nd

to C

SU

)


N= 671,861 Effect =.003 (.004)

Probability of Attending a CSU by EAP English Score

Early negative information does not discourage students from enrolling

• The English readiness signal has no impact for EAP Math participants

• For math non-participants:

– There is a small negative impact on application to a CSU

– There is no impact on CSU enrollment

35


36


Students are no more likely to enroll in advanced math when receiving conditional exempt signal

37


38


02

04

06

0

# o

f S

cho

ols

0 20 40 60 80 100% Participation in English EAP

Using EAP participation quartiles

Percent of students taking EAP English Exam per school (2005 Cohort)

39

Higher EAP participation is associated with higher levels of CST Proficiency

1.8

0

10

20

30

40

50

Q1 Q2 Q3 Q4

% P

ro

fic

ien

t o

f A

dv

an

ce

d

Pre-EAP Post-EAP

*

Fitted Values for CST Proficiency from Difference in Difference

Pre-Post Differences in School-wide CST Proficiency by Participation Quartiles

40

0

2

4

6

8

10

12

14

16

18

Q1 Q2 Q3 Q4

% A

dv

an

ce

d

Pre-EAP Post-EAP

Higher EAP participation is associated with higher levels of CST Advanced

Fitted Values for CST Advanced from Difference in Difference

1.3

*

*

Pre-Post Differences in School-wide CST Advanced by Participation Quartiles

41

200

300

400

500

600

700

800

Q1 Q2 Q3 Q4

AP

I

Pre-EAP Post-EAP

Higher EAP participation is associated with higher API

Fitted Values for API from Difference in Difference

5.3

*

Pre-Post Differences in School-wide API by Participation Quartiles

42

Conclusions: EAP and college readiness at high schools

• School characteristics are mostly unrelated to EAP participation

• Schools in all levels of participation see gains in test scores, accountability measures, and college-level outcomes

• Higher levels of participation in EAP are associated with higher gains across tests scores and school accountability measures

43


44


Very Few CCC students are exempt via

the EAP

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

English Math

Pe

rc

en

t o

f a

ll E

nte

rin

g S

tud

en

ts

Percent of CCC Entering Students Scoring Exempt in 2009

Eligible Take EAP Conditional Exempt Exempt

45

EAP Exempt students take more demanding courses

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CSU Transferable UC Transferable Basic Remedial

Pe

rc

en

t o

f C

las

se

s

Percent of Transferrable Classes for EAP Exempt Students

ALL Students English exempt Math Exempt

46

EAP at Community Colleges

47

40%

45%

50%

55%

60%

65%

70%

75%

80%

1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013

Pe

rc

en

t o

f F

irs

t-T

ime

Stu

de

nts

English Developmental Course Participation

Early Adopters

Late Adopters

No EAP

Summary: Unintended Consequences of the EAP

• The EAP does not dissuade students from applying to or enrolling in a CSU

• The EAP does not impact student 12th grade course-taking

• EAP participation may benefit high school accountability measures

• EAP adoption at community college is associated with lower rates of developmental coursetaking

48

FUTURE RESEARCH ON THE EAP AND COLLEGE READINESS

Section 4

49

Future work on college readiness

• Long term outcomes of EAP

– Graduation

– Time to degree

– Persistence

• College Readiness in the era of Common Core and Smarter Balanced Assessments

• Early Start at California State University

50

[email protected] [email protected]

51

Documents

Program: Assessing Intended and Unintended … eap...2 4 6 8 10 12 14 16 18 Q1 Q2 Q3 Q4 d Pre-EAP Post-EAP Higher EAP participation is associated with higher levels of CST Advanced