LEARNING FROM EXPERIMENTATION IN DEVELOPMENTAL MATH IN COMMUNITY COLLEGES IN CALIFORNIA: RESULTS FROM A LONG-TERM RESEARCH PARTNERSHIP

The Steinhardt Institute for Higher Education Policy,

New York University, October 19, 2015

Tatiana MelguizoAssociate Professor, University of Southern Californiamelguizo@usc.edu

This research was funded by a grant from the U.S. Department of Education’s Institute of Education Sciences (IES).

Overview

Educational & professional background Key results from a federally funded

long-term research collaboration with a large urban community college district in California

Other research interests Q & A

Inequalities in college access and outcomes by socioeconomic status in Colombia

Only 25% of students from the lowest income strata enter college compared to 60% of students from the highest income strata(Melguizo, Sanchez & Velasco, 2015)

Los Andes University Bogota-Colombia

Increasing inequalities in educational outcomes in the U.S.

Banning of Affirmative Action Policies in the U.S. in the mid 1990s

I started my Ph.D. at the time that Affirmative Action policies were banned in California

That major policy shift motivated me to study the impact of attending more selective colleges and universities for students of color

Findings suggested that students of color both in the 1980s and in the 1990s had higher graduation rates at the most selective colleges and universities (Melguizo, 2007)

Findings cited as part of Amicus Brief to the Supreme Court by AERA

Creating a Research Partnership to Study Assessment and Placement in Developmental Math

The purpose of the partnership is to inform developmental education research, policies, and practices Site: The Los Angeles Community College District Research Focus: To examine the impacts and

implementation of test-based, alternative placement policies, and delivery methods for developmental math

Research approaches: Experimental and quasi-experimental, qualitative, & descriptive

Strength of partnership built on commitment & unique expertise of partners A joint commitment to improve achievement in

dev. ed. through policy and practical change

An firm belief that each partner is an equal; brings unique contextual, technical expertise LACCD: Knowledge about local context, access to

administrative records, faculty, and administrators

USC: A strong record of community college research

AIR: Technical expertise in conducting experimental and quasi-experimental research

Problem Statement

• Every year about 80 percent of community college students in California are placed into preparatory mathematics. This percentage is higher than the national average

• Community college students have widely varying initial skills levels

• Colleges have to offer classes to meet these levels and have to keep heterogeneity in the classrooms manageable

• Placing students incorrectly can reduce the likelihood that students succeed

Why LACCD?

Los Angeles Community College District - a natural laboratory

Diverse student population that varies by college.

Nine colleges with 130,000 plus students. “Common data system.” Large number of observations. Presumption of representativeness—likely

to capture the wide variation across community colleges in the United States.

Literature on Impacts of Remedial Education

• Proponents argue that remedial education provides the preparation necessary for students to succeed in college (Boylan, Bliss, & Bonham, 1994; 1997; Lazarik, 1997)

• Critics contend that the benefits that students obtain are not clear (Calcagno & Long, 2008; Martorell & McFarland, 2011; Scott-Clayton & Rodriguez, 2015).

Developmental Math Sequence

Developmental Math

Remediation needs of LACCD students

Five Key FindingsFinding 1: Establishing an effective A&P system is complex.

More support and training is needed for faculty and administrators charged with this task. (Melguizo, Kosiewicz, Prather & Bos, 2014).Finding 2: The largest barrier for developmental math students is attempting their initial course (Fong, Melguizo, & Prather, 2015).

Finding 3: Community college faculty and administrators have the opportunity to improve placement and success in developmental math by engaging in a systematic process of calibration of the cut scores of assessment and placement tests (Melguizo, Bos, Ngo, Mills & Prather, 2015).

Finding 4: The diagnostic test places students more accurately than the computer-adaptive test (Ngo & Melguizo, 2015).

Finding 5: The inclusion of multiple measures in the placement process can increase access to higher-level math without decreasing students’ chances of success (Ngo & Kwon, 2014; Fong & Melguizo, 2015).

F2: Over 30 percent of students are NOT attempting the assigned courses

F3: Systematic Process of Calibration of the Cut Scores

Math faculty set the cut points between the different levels based on who applies and how their course offerings are distributed

If the cut points are too high, too many students languish in remedial courses

If the cut points are too low, too many students fail higher-level courses and present a challenge to the instructors

Getting the cut points just right is important

F3: Different Pathways to Success(Arithmetic vs. Pre Algebra)

Test

Placed in Pre-Algebra

Placed in Arithmetic

Enroll in Pre-Algebra

Enroll in Arithmetic

No enrollment

Success

Failure

Success

Failure

Next course

F3: Ideal Regression Discontinuity Situation

Regression discontinuity analysis is the strongest non-experimental method to estimate causal effects

It depends on a continuous forcing variable and an exogenously established cut point

Those two conditions are present in this situation

Out

com

e

Placement Test Score

Regression Line

Arithmetic

Pre-Algebra

Imp

act

F3: Negative Impact of Starting in ARTime Dimensions is Key

F4: Placement Accuracy:Diagnostic versus Computer-AdaptiveTestsDo diagnostics improve placement accuracy?MethodsI.Logistic Regressions

Predicting EA outcomes with skill-specific math information

II.Placement Accuracy Sum of accurate placements

III.Regression Discontinuity Traditional (with single-score) Binding-score (with multiple criteria)

Placement AccuracyTable 3. Percent of accurate placements by level of developmental math.

AR vs PA PA vs EA EA vs IA

ACCUPLACER Colleges      

College A 73.2 75.3 72.9

College B - 75.5 48.4

College C 19.8 22.6 24

College D 77.8 47.2 71.4

MDTP Colleges

College G (w/subscores)

51.6 76.3 56.1

College G (w/o subscores)

49.9 68.2 44.9

College H 90.0 65.5 54.1Note: Placement accuracy calculated using method described by Scott-Clayton (2012). Percentage shown is the sum of the proportion of students predicted to pass higher-level course and placed there plus the proportion of students not predicted to pass the higher-level course and placed in the lower level course.

F4: Diagnostic tests are Placing Students more Accurately than Computer-Adaptive Tests

Students placed using results from computer-adaptive tests were more negatively impacted by the placement decision than prior cohorts placed by MDTP.

Students were less likely to enroll and persist onto the next math course after the placement test switch.

Consistent with other studies, we found that the diagnostic test can provide information on student proficiency on a range of subtopics such as fractions, exponents, and reasoning which can improve math placement decisions and/or tailor instruction in math courses.

F5: The Inclusion of Multiple Measures can Increase Access w/out Decreasing Student Success

F5: Findings

Only 6% of the students benefitted from multiple measures at the LACCD

Major benefits for African American and Latino students who could enroll in higher-level math courses

We found no evidence that “boosted” students were less likely to complete the course

Conclusions

Engaging in research partnerships with large districts are a way of increasing research capacity of the district while gaining a much more nuanced understanding of the context for the researcher

Research findings not only contribute to the knowledge but are also policy relevant and actionable

Researcher-practitioner partnerships can be conducive to high-quality and high-impact research

Other Current Projects Mixed-methods evaluation of the Susan

Thompson Learning Communities. Buffett Foundation

Using high school transcript information to refine A&P in math. LAUSD-LACCD-USCNSF: EAGER

Accurately estimating Student Learning Outcomes (SLOs) in higher education Brazil (Melguizo & Wainer, under review) Colombia (Melguizo, Zamarro, Sanchez & Velasco,

2015) Proposed an RCT evaluation to test student

self-placement in Dev Ed Math

THANK YOU!Questions

Tatiana Melguizomelguizo@usc.edu

http://www.uscrossier.org/pullias/research/projects/sc-community-college/