Education, Evidence, and Policy

Cecilia Elena Rouse, Ph.D. Member of the Council of Economic Advisers

June 8, 2009

The Council of Economic Advisers was established by Congress in the Employment

Act of 1946:

“There is hereby created in the Executive Office of the President a Council of Economic Advisers (hereinafter called the “Council”). The Council shall be composed of three members who shall be appointed … to appraise programs and activities of the Government …and to formulate and recommend national economic policy to promote employment, production, and purchasing power under free competitive enterprise.”

It shall be the duty and function of the Council --

• to assist and advise the President in the preparation of the Economic Report;

• to gather timely and authoritative information concerning economic developments and economic trends….;

• to appraise the various programs and activities of the Federal Government….;

• to develop and recommend to the President national economic policies to foster and promote free competitive enterprise, to avoid economic fluctuations or to diminish the effects thereof, and to maintain employment, production, and purchasing power;

• to make and furnish such studies, reports thereon, and recommendations….

Educators“Scientific Learning's family of programs and products are for anyone who wants to improve language, reading and overall communication skills. From children who struggle with basic language skills or attention problems to adults who want to improve reading comprehension and organizational skills, Scientific Learning's programsand products have something for everyone.”

• 4 schools in a large urban school district in the Northeast

• Eligibility based on low reading score on Fall 2001 State Assessment (Bottom 20%)

• 513 students in grades 3-6• Random assignment within grade and school• Administered CELF-3 and Reading Edge Test;

SFA Assessments

Rouse, Krueger, and Markman Evaluation of Fast ForWord

0

5

10

15

20

25

30

35

pre post

CE

LF-3

NC

E (M

ean=

50, S

D=2

1)

CELF-3 Test: Gains Among FFW Participants

N = 43 Effect size=0.3σ p-value=0.008

0

10

20

30

40

50

60

70

80

pre post

Co

mp

osi

te R

.E.

Sco

re (

SD

=30

)

Reading Edge Test: Gains Among FFW Participants

N = 244 Effect size=0.7σ p-value=0.000

But, what would have happened had the children not participated in FFW

(i.e., what’s the counterfactual)?

0

5

10

15

20

25

30

35

Treatments Controls

CE

LF

-3 N

CE

(M

ean

=50

, S

D=

21)

pre

post

CELF-3 Test:Gains Among FFW Participants and Controls

N = 89 “Net” Effect size=0.04σ p-value=0.73

SLC’s Reading Edge Test:Gains Among FFW Participants and Controls

40

45

50

55

60

65

70

75

Treatments Controls

Com

posi

te R

.E. S

core

(SD

=30)

pre

post

N = 485 “Net” Effect size=0.13σ p-value=0.098

3.7

3.8

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

FFW Participants Experimental Control GroupPo

st S

FA

Ass

essm

ent

(Mea

n=3

.7, S

D=1

.5)

Post FFW SFA Assessment: FFW Participants vs. Experimental Controls

N = 197 FFW Participants; 176 Experimental ControlsEffect size=0.05σ p-value=0.583

3.7

3.8

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

FFW Participants ExperimentalControl Group

Non-experimentalComparison Group

Po

st S

FA

Ass

essm

ent

(Mea

n=3

.7,

SD

=1.5

)

Post FFW SFA Assessment: FFW Participants vs. Non-experimental and

Experimental Controls

Experimental effect size=0.05σ (p-value=0.583) Non-experimental effect size=-0.3σ (p-value=0.000)

N = 197 FFW Participants; 3850 Non-experimental Comparisons; 176 Experimental Controls

0

0.01

0.02

0.03

0.04

0.05

0.06

Non-experimental EstimatedEffect

Experimental EstimatedEffect

Est

imat

ed E

ffec

t o

f F

FW

Non-experimental vs. Experimental Estimated Effects of FFW on Gain in SFA Assessments

Non-experimental: N=4048 “Net” Effect size=0.04σ p-value=0.11 Experimental: N=373 “Net” Effect size=0.01σ p-value=0.60

Example of Results that Led to Push for “College for All”

0

0.5

1

1.5

2

2.5

3

3.5

Non-College Prep. (GPA) College Prep. (GPA) Non-College Prep. (IRT) College Prep. (IRT)

Gra

de

-Po

int

Av

era

ge

0

10

20

30

40

50

60

70

IRT

- e

sti

ma

ted

ma

th s

co

re

First-Year GPA Final-Year IRT-estimated math

Source: Valerie Lee, et al (1998), "High School Curriculum Structure: Effects on Coursetaking and Achievement in Mathematics for High School Graduates"

**Statistically significant at the p=.05 level.

1.38 GPA**33.73**

Chicago’s “College Prep for All”

• Instituted in 1997• Requires all students to take:

1. Four years of English + Three years of Math, science, and social science

2. Particular classes: Algebra I, geometry, Algebra II (math); survey literature, American literature, European literature, world literature (English); biology, earth science, and chemistry or physics (science); world studies, U.S. history, and one elective (social science)

First Results from Chicago’s “College Prep for All” (Mathematics)

Pre-Policy Post-Policy Pre-Policy Post-Policy

Gra

de

-Po

int

Av

era

ge

First-Year GPA Final-Year GPA

Source: Elaine Allensw orth (2008), "College Preparatory Curriculum for All: Consequences for Ninth-Grade Course Taking in Algebra and English on Academic Outcomes in Chicago"

Note: Differences are not statistically significant at the p=0.10 level.

Difference = 0.04 GPA

Difference = 0.01 GPA

Limits of most “rigorous” studies for policy:

• Heterogenous treatment effects;• Size of likely impacts are difficult to

interpret/translate;• Difficulty or ease of bringing to scale;• Typically doesn’t illuminate the mechanism

by which a particular program/policy works;

• Longer-run impacts are usually difficult because of cost (especially time).

So Where Does this Leave Us?

• Current evidence-base for most education programs/reforms is extremely thin.

• Need more high-quality impact evaluations to help guide everyday decisions.

• It’ll take cooperation from all to improve the quality of education research on impacts.