#NACAC17

B12:Testing, Academic Achievement, and the Future of College Admissions

#NACAC17

Measuring Success:Testing, Grades, and the Future of College Admissions

Jack BuckleyAmerican Institutes for Research, Washington DC

#NACAC17

Measuring Success: Two Key Questions

• More heat than light in the debate over standardized testing in admissions

• My co-editors and I wanted to produce a volume that brings together the best

empirical thinking around two questions:

– To what extent should standardized tests play a role in college and university

admissions?

– How should an institution of higher education best assess prospective student talent in

a manner that promotes fairness?

• Result is the forthcoming volume from Johns Hopkins University Press!

#NACAC17

Overview

• Chapter 1: Eight Myths about Standardized Admissions Testing

– Paul Sackett and Nathan Kuncel, University of Minnesota

• Chapter 2: The Core Case for Testing: The State of Our Research Knowledge

– Emily Shaw, The College Board

• Chapter 3: Grade Inflation and the Role of Standardized Testing

– Michael Hurwitz, The College Board and Jason Lee, Tennessee Higher Education Commission

• Chapter 4: Merit-Based Scholarships in Student Recruitment and the Role of Standardized Tests

– Jonathan Jacobs, Jim Brooks, and Roger Thompson, University of Oregon

#NACAC17

Overview

• Chapter 5: When HSGPA and Test Scores Disagree: Implications for Test-

Optional Policies

– Edgar Sanchez and Krista Mattern, ACT

• Chapter 6: The Rise of Test-Optional Admissions Practices

– Jerome Lucido, University of Southern California

• Chapter 7: Going Test-Optional: A Case Study

– Eric Maguire, Franklin & Marshall College

• Chapter 8: Test Scores and High School Grades as Predictors

– William Bowen (Mellon), Matthew Chingos (Urban), Michael McPherson (Spencer)

#NACAC17

Overview

• Chapter 9: How Do Percent Plans and Other Test-Optional Admissions Programs

Affect the Academic Performance and Diversity of the Entering Class?

– Rebecca Zwick, Educational Testing Service

• Chapter 10: The Test-Optional Movement at America’s Selective Liberal Arts

Colleges: A Book for Equity or Something Else?

– Andrew Belasco (College Transitions), Kelly Rosinger (Penn State), and James Hearn

(University of Georgia)

• Chapter 11: The Effect of Going Test-Optional on Diversity and Admissions: A

Propensity Score Matching Analysis

– Kyle Sweitzer, A. Emiko Blalock, and Dhruv Sharma, Michigan State University

#NACAC17

Teaser from Chapter 3

#NACAC17

When HSGPA and Test Scores Disagree: Implications for Test-Optional Policies

Edgar Sanchez & Krista MatternACT, Iowa

#NACAC17

Test Optional Policies

• Test-optional policies come in many shapes and sizes– No standardized test scores are considered for any applicant

– Standardized test scores are still required, but applicants may choose from a list that includes the ACT and SAT and various tests such as AP, IB, or SAT II subject tests.

– Standardized test scores may or may not be required depending upon other criteria such as program being applied to, home-schooling, out-of-state residency, or HSGPA or class rank minimum

– Placement tests or school-specific admissions exams rather than ACT/SAT

– Standardized test used only for placement

– Even colleges that offer to consider an application without test scores may require additional proof of achievement, such as a graded writing sample or an on-campus interview

#NACAC17

Perceived Benefits of Test Optional Policies

• Address perceived concerns with Standardized Testing– Improper use of tests

– Equity in access to test prep/coaching

– Fairness/Bias

– Predictive validity of test scores

• Increase diversity

#NACAC17

Researched Responses to Perceived Benefits

• Little scientific research to suggest prep/coaching has a significant impact on test scores. Hard work in school and rigorous course taking has a strong impact on test scores.

• Gaps cited as concerns in test scores are also seen in all other academic achievement indicators including HSGPA. There are many studies that document the subgroup gaps, but rather than highlight bias they seem to highlight population differences resulting from educational experiences.

• There are many studies that highlight the predictive power of test scores. While HSGPA is typically the strongest predictor of postsecondary outcomes, test scores provide unique and valuable additional information.

• Research has shown mixed results in regards to diversity outcomes. Additionally, test optional policies are often implemented in conjunction with other awareness and recruitment initiatives making it difficult to isolate the effect of test optional policies on campus diversity.

#NACAC17

Discrepant ACT and HSGPA

• Prior research has documented the relationship between academic preparation, as measured by high school grade point average (HSGPA) and admissions test scores, and successful college outcomes.

• Test scores and grades often send a consistent message about the academic achievement level of students.

• There are instances, however, when test scores and grades send mixed messages about the academic performance of students.

#NACAC17

Heat map of ACT Composite score and HSGPA for the ACT-tested graduating class of 2016 (N= 1,709,659)

Relationship between ACT Composite and HSGPA

#NACAC17

Implications to Test Optional Colleges

• The impact of discrepant performance on college outcomes is mixed making an even stronger case for considering both pieces of information when considering an applicants' likelihood of future success.

• More precise diagnoses of a student's level of college readiness is useful not only for admissions but also for providing tailored interventions for at-risk students.

#NACAC17

Implications to Test Optional Colleges

• Compensatory nature of test scores and HSGPA– Higher grades can offset lower test scores and vice versa in terms of future

college outcomes.– High school grades encompass students’ non-cognitive factors such as

scholastic engagement, self-regulation, discipline, or habits of inquiry, in addition to providing information on cognitive skills.

– Standardized tests, on the other hand, provide a more narrow assessment focusing on cognitive skills in a standardized manner.

• This compensatory relationship is lost when institutions elect to institute test-optional policies.

#NACAC17

• Multiple pieces of information presents a clearer picture of students’ strengths and weaknesses and their likelihood of future success emerges.

• Empirical evidence of the validity of test scores for use in college admissions underscores the value of multiple measures.

– This is particularly true when the various pieces of information are inconsistent. These are the very students who are mostly likely to be attracted to test-optional institutions.

• While test-optional policies promote discarding useful information, we stand behind the belief that more information is better than less information.

Implications to Test Optional Colleges

#NACAC17

0.90

0.82

0.51

0.87

0.77

0.69

0.88

0.69 0.67

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Pr

obab

ility

of E

nrol

lmen

t

Example: Probability of Enrollment by Discrepancy Group

#NACAC17

Example: Probability of Persistence by Discrepancy Group0.81

0.69

0.57

0.73

0.60 0.60

0.76

0.670.65

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Prob

abili

ty o

f Per

sist

ence

#NACAC17

The Rise and Relevance of Test Optional Admission Policies

Jerome A. Lucido, Professor and Exec. Director

USC Center for Enrollment Research, Policy, and Practice

#NACAC17

Charge for the Chapter

Analysis of test-optional from a practitioner/scholar:

• What are its stated purposes and claims

• What does research on the matter reveal

• Structured interviews

• Policy analysis

#NACAC17

Guidelines for Policy and Practice

• Commentary on the way forward for those who wish to:

- Consider policy options,

- Examine their results, and

- Draw upon the latest thinking in this realm

#NACAC17

Guidance for the Profession

• Places test-optional practices within the evolution of the

admission and enrollment management profession

• Examines the role of testing and test-optional policies in

promoting college access.

• Considers the potential of both to further influence thinking

and practice.

#NACAC17

A Very Brief History of Test-Optional

• Bowdoin (1969)

• Atkinson and the University of California (2001)

• Hiss and Franks at Bates (2004) (2014)

• NACAC Commission: Report on the Use of Standardized

Tests in Undergraduate Admission (2008)

• Foundation in liberal arts colleges

#NACAC17

Purposes and Objectives of T-O

• Concerns about testing and the by-products of testing:

- Misuse or narrow use of results in admissions

- Test prep industry

- Stress of testing

- Concerns about efficacy for underrepresented

populations

#NACAC17

• Student Benefit

- Addresses fears

- Opens minds to possibility

- Brings rhetoric into balance with practice

Purposes and Objectives of Test-Optional

#NACAC17

• Institutional Benefit

- Enrollment and rankings strategy

- Diversity enhancement

- T-O does not operate in a vacuum

Purposes and Objectives of Test-Optional

#NACAC17

• Conflicting Research

- Belasco, et al

- Hiss and Franks

- The criticality of local studies

How Do T-O Policies Play Out in Practice?

#NACAC17

• Self limiting application decisions

• Narrow admission heuristics

• Overreliance on profile

• Discouragement of underrepresented prospects

• The intimidation factor

Barriers Addressed by Test-Optional

#NACAC17

• A view of the test-optional adopter as

pragmatic idealist emerges from this

investigation

• Not actually test “opponents”

The People Behind Test-Optional

#NACAC17

• Neither panacea nor the only way

• The future is not one or the other

• Toward enlightened practice

• No substitute for expertise and information

Reconciling Testing and Test-Optional

#NACAC17

• While T-O seeks to mitigate negative impacts of

testing and test misuse (among other things), the

need for sound information in the admission

decision remains.

Conclusion

#NACAC17

• Test results, in the hands of thoughtful practitioners,

adds value to admission decisions. Can wise

decisions be made without them? Years of test-

optional practice now suggests that they can.

Whether they “should” is a matter of institutional

choice, mission, and local research.

Conclusion

#NACAC17

• Testing and T-O together hold the seeds of better

and more highly informed practice. This includes

mitigating the negative impact of test shortcomings

and misuse while enlightening admission decision

makers with information that can lead to better

admission decisions, better academic advising, and

better results for students.

Conclusion

#NACAC17

• Jerome A. Lucido, Professor and Director

• USC Center for Enrollment Research, Policy and

Practice

jlucido@usc.edu

http://cerpp.usc.edu/

Thank You!