Interpreting the Evidence around College Access Strategies Neal D. Finkelstein Society for Research...

Interpreting the Evidence around College Access

Strategies

Neal D. FinkelsteinSociety for Research on Educational Effectiveness

March 4, 2010

1

Background• IES Practice

Guide• “Helping

Students Navigate the Path to College: What High Schools Can Do”

• September 2009

2

Background

• What Works Clearinghouse standards were used to determine levels of evidence – strong, moderate and low.

3

Panel Recommendations

4

1) Offer courses and curricula that prepare students for college-level work, and ensure that students understand what constitutes a college-ready curriculum by 9th grade.

Level of Evidence: Low

Panel Recommendations

5

2) Utilize assessment measures throughout high school so that students are aware of how prepared they are for college, and assist them in overcoming deficiencies as they are identified.

Level of Evidence: Low

Panel Recommendations

6

3) Surround students with adults and peers who build and support their college going aspirations.

Level of Evidence: Low

Panel Recommendations

7

4) Engage and assist students in completing critical steps for college entry.

Level of Evidence: Moderate

Panel Recommendations

8

5) Increase families’ financial awareness, and help students apply for financial aid.

Level of Evidence: Moderate

Research Scan

• Research scan over the period 1988-2008 revealed more than 500 studies

• 99 studies had causal designs• 16 studies of 10 programs met

WWC standards, with or without reservations

9

Research Scan

• The summary of evidence is methodical, but not without complexity. For example, 13 studies of programs potentially met WWC standards, but provided insufficient information to establish baseline equivalence.

10

Research Scan

• No easy wins on the strict mapping of evidence to recommendations; the benefit to the practitioner lies in panelists working through the subtleties.

11

Research Scan

“College access programs consist of multiple components that address a variety of steps students must take to prepare for and enter college. The bundling of multiple practices within an access program makes it difficult to assess the effectiveness of a single practice.” (Page 49)

12

Research Scan

• Positive effects on one or more relevant outcomes: 5 programs.

• No detectable effects on relevant outcomes: 5 programs.

13

Research Scan

14

Positive Effects No detectable Effects

FAFSA Experiment (2009) EXCEL (2007)

Career Beginnings(1990) Middle College High Schools (1998)

Talent Search (2006) Career Academies (2000-2008)

Sponsor-a-Scholar (1998) Quantum Opportunity Program (QOP)

Talent Development High School (2005)

Upward Bound (1997-2009)

Observations

• The glass is ¼ full – some evidence with some caveats.

• The institutional context of program implementation sites challenges generalizability.

• The research standards by the WWC set the bar for new studies – essential for the long term.

15

Observations

Assertion: College access programs implement as bundles of interlocking strategies, and schools operate as settings for interlocking strategies.

16

Challenge

Design a study, which meets rigorous standards, that can differentiate the effectiveness of individual strategies, within the context of the real-time school setting.

17

Interpreting the Evidence around College Access

Strategies

Neal D. FinkelsteinSociety for Research on Educational Effectiveness

March 4, 2010

18