Embed Size (px)
Citation preview
Understanding Standardized Testing: A Deeper Look
1
Developed more than 80 years ago as a tool to help democratize higher education access for all students, the SAT
® has grown to become the world’s most widely used college entrance exam.
The SAT continues to evolve and improve to meet the needs of a dynamic education landscape. However, the SAT has always served the primary mission of the College Board: to connect students to college opportunity and success.
History and Background
4
A Measure of College Readiness
The SAT evaluates the cognitive tools necessary to succeed in college and beyond, including the ability to:
– Think critically
– Solve problems
– Communicate effectively
What is the SAT®
?
5
A Predictor of College Outcomes
Is a valid predictor of meaningful college outcomes on a student’s path to a college degree, including:
– Freshman Year GPA
– 2nd, 3rd and 4th year GPAs
– College Retention
What’s On The Test…
The SAT® assesses the core academic skills necessary for college success and how students apply those skills.
6
44 multiple-choice items, 10 student-produced response items
Number & Operations, Algebra, Geometry, Data Analysis
25 minute essay, take sides on an accessible topic
49 multiple-choice grammar/usage questions
Essay sub-score 2-12
Short (paragraph) and long (up to 800 word) passages
48 questions, make inferences, draw conclusions
19 sentence completion questions
Critical Reading200 - 800
Mathematics200 - 800
Writing200 - 800
Every SAT® Knowledge and Skills Topic is represented in the Common Core State Standards
The knowledge and skills covered on the SAT are directly linked to what students are learning in the classroom
7
SAT Critical Reading Knowledge and Skills Topics
SAT Mathematics Knowledge and Skills Topics
SAT Writing Knowledge and Skills Topics
Number & Operations
Algebra & Functions
Geometry & Measurement
Data, Statistics & Probability
Problem Solving
Representation
Connections
Communication
Manage Word Choice and Grammatical Relationships Between Words
Manage Grammatical Structures Used to Modify or Compare
Manage Phrases and Clauses in a Sentence
Recognize Correctly Formed Sentences
Manage Order and Relationships of Sentences and Paragraphs
Determining the Meaning of Words
Author’s Craft
Reasoning and Inference
Organization and Ideas
Understanding Literary Elements
Source: Vasavada, N., Carman, E., Hart, B. Luisier, D.; Common Core State Standards Alignment: readiStep™, PSAT/NMSQT® and SAT; College Board, 2011
Students enrolled in a core curriculum perform better on the SAT®
Core curriculum is defined by at least four years of English, and at least three years of mathematics, three years of natural science, and three years of social science and history.
+48 points
+47 points
+48 points
Scor
e
Source: College-Bound Seniors 2011 Cohort Data8
SAT® Mean Scores by Curriculum
Students who take a core curriculum in high school perform better on the SAT and are better prepared for college than students who do not.
Students enrolled in rigorous courses perform stronger on the SAT ®
9
+59 points +46 points
+58 points
Scor
e
Source: College-Bound Seniors 2011 Cohort Data
SAT Mean Scores by AP or Honors English Participation
Students enrolled in AP ® or Honors English outperform the general SAT populations in all sections of the SAT.
Students enrolled in rigorous courses perform stronger on the SAT ®
Source: College-Bound Seniors 2011 Cohort Data10
+64 points
+76 points
+64 points
Scor
e
SAT Mean Scores by AP or Honors Math Participation
Students enrolled in AP ® or Honors Mathematics outperform the general SAT populations in all sections of the SAT.
1991 1996 2001 2006 2011
Grade Average for All Subjects* 3.10 3.20 3.28 3.33 3.34
High school grades are increasing over time
*Based on four-point system, where A=4.0Note: 1990 GPAs reflect both SAT Subject Test™ takers and SAT® takers. GPAs for 1995-2010 reflect SAT® takers only.
11
The need for a consistent national measure is more important than ever as high school grades have been increasing over time.
Percentage of Students by Self-Reported High School GPA
West Ed Alignment | Summary of Findings
Competency SAT Alignment ACT Alignment
UC – Academic Literacy
Critical Reading: 90% Strong, 10% Implicit (B8)
Writing: 100% Strong (B8)
Essay: 3 standards aligned
Reading: 85% Strong, 13% Implicit, 2% None
English: 100% Strong
Essay: 3 standards aligned
UC – Mathematics
Mathematics: 85% Strong, 13% Partial, 2% Implicit Mathematics: 82% Strong, 12% Partial
CA – English Language Arts (Grades 9-10)
Critical Reading: 66% Strong, 34% None (B8)
Writing: 100% Strong (B8)
Essay: 5 standards aligned
Reading: 25% Strong, 8% Partial, 67% None
English: 100% Strong
Essay: 5 standards aligned
CA – English Language Arts (Grades 11-12)
Critical Reading: 37% Strong, 63% None (B1)
Writing: 98% Strong, 2% None (B1)
Essay: 5 standards aligned
Reading: 40% Strong, 60% None
English: 97% Strong, 3% None
Essay: 5 standards aligned
CA – Mathematics
Mathematics: 63% Strong, 35% Partial, 2% None
SAT II 3YBC: 82% Strong, 12% Partial, 6% None
Mathematics: 77% Strong, 22% Partial, 1% None
CA – Science SAT II Physics: 57% Strong, 23% Partial, 20% None
SAT II Chemistry: 58% Strong, 32% Partial, 10% None
SAT II Biology: 39% Strong, 17% Partial, 44% None
Science: 0% Strong, 3% Partial, 97% None
*All information taken from WestEd SAT/ACT Alignment Study
SAT® Participation
13
More students are planning for college as evidenced by the increase in SAT ® participation
Source: 2011 College-Bound Seniors Total Group Report; WICHE (2008)14
1,534,4571,563,272 1,573,110
1,597,329
1,647,123
2007 2008 2009 2010 20110%
1%
2%
3%
4%
5%
6%
2007 2008 2009 2010 2011
% C
hang
e vs
. 200
7 Co
hort
US SAT TakersUS High School
Graduates
SAT Participation Relative to U.S. High School GraduatesSAT Participation by Cohort
The SAT reaches more students than ever before while the number of graduating high school seniors in the U.S. has decreased.
The SAT® Is reaching more underserved students
2011 SAT Takers Beneficiaries of the SAT Fee-Waiver Program
545,010
533,731
507,782
483,842
487,113
2011
2010
2009
2008
2007
2011 SAT Takers Prospective First-Generation College Goers
15
351,068
309,289
269,015
230,080
198,729
2011
2010
2009
2008
2007
Source: 2011 College-Bound Seniors
77% increase in SAT Fee-Waiver usage since 2007
Represents more than $37 million in fees and services
The SAT is reaching more low-income and first-generation students who are traditionally underserved in the college-going process.
The SAT ® is the most diverse U.S. college entrance exam
Note: Due to rounding, percentages do not add up to 100.
Source: 2011 College-Bound Seniors 16
11%
Asian
13%
Black
15%
Hispanic/Latino
4%
Other
53%
White
4%
No Response
1%
American Indian
2011 College-Bound Seniors by Race/Ethnicity
43% of SAT takers report an ethnic or racial background other than Caucasian
Minority SAT ® takers are increasing at a faster rate than U.S. public high school grads
17
The growth rate of SAT takers is highest for racial and ethnic minority sub-groups
Source: 2011 College-Bound Seniors Total Group Report; WICHE (2008)
SAT® reflects the diversity of the nation’s classrooms
US Public SchoolSAT Takers
US Public SchoolGraduates
84% of SAT Takers report attending public school
18Source: 2011 College-Bound Seniors Total Group Report; WICHE (2008)
SAT participation closely reflects the distribution of minority students graduating from U.S. public schools.
19
SAT ® Validity
20
Why is Predictive Validity important?
Predictive Validity refers to the ability of a factor (e.g. test scores) to predict future performance.
Predictive Validity refers to the ability of a factor (e.g. test scores) to predict future performance.
For colleges and universities, it is important to understand how well factors used for college admissions or placement predict desired academic outcomes
– Factors (predictors) - typically include high school grades, SAT ® scores or class rank
– Academic Outcomes – GPA, retention, course grades
The measurement of how well predictors do is usually expressed as a correlation (from +1.0 to -1.0)
22
National SAT ® Validity Study
Cross-institutional, longitudinal validity and higher education research informing ways to ensure that students are ready for and successful in college.
* College Board Research and Development Database, 2007 Cohort
National SAT Validity Study*
Data supplied by four-year institutions from around the U.S. and matched to College Board data.
•110 institutions•216,081 students
Broad institutional characteristics•Size•Public/Private•Geographic distribution•Selectivity
The SAT ® is a strong predictor of first year college performance
Correlation* of SAT® and High School GPA to First-Year College
GPA
Each section is a valid and strong predictor of college performance
* Correlations corrected for restriction of rangeSource: Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting FYGPA: 2007 SAT Validity
Sample; College Board, 2011 23
The combined SAT predicts as well as high school GPA
The SAT in combination with HSGPA is the best predictor of college performance
The SAT continues to predict just as well as high school grades. When used together, grades and SAT scores are the best predictors of college performance
What does a correlation of .56 mean?
The SAT® provides a meaningful prediction of how students will perform in their first year of college.
8%18%
33%
54%
74%
89%
0%
20%
40%
60%
80%
100%
600-890 900-1190 1200-1490 1500-1790 1800-2090 2100-2400
% E
arni
ng B
or
high
er
Freshman GPA of B or Higher
SAT Scores: Mathematics + Critical Reading + Writing
24Source: Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting FYGPA: 2007 SAT Validity
Sample; College Board, 2011
Other Correlation Examples for Context
Variable 1 Variable 2 Correlation Sugar Consumption
Child’s Behavior .00
Aspirin Heart Attack Death .02
Lead Exposure Child IQ .12
Parental Divorce Child Well-Being .09
Ibuprofen Pain Reduction .14
Alcohol Aggressive Behavior
.23
Viagra Sexual Functioning .38
Gender Height .67
SAT (CR, M, W
) with
FYGPA ≈ .56
From: Meyer, G., et. al. (2001). Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist, 56, 128-165.
SAT ® Scores Still Matter, Even AfterControlling for High School GPA
27
Mean FYGPA by SAT Score Band, Controlling for HSGPA
SAT 600 - 890 900 - 1190 1200 - 14901500 - 1790 1800 - 2090 2100 - 2400
2.79
3.09
3.34
3.56
2.22
2.522.51
2.74
2.94
3.15
1.98
2.262.12
2.27
2.472.54
1.81
1.98
FYG
PA
1.0
1.5
2.0
2.5
3.0
3.5
4.0
HSGPAC or Lower B A
The SAT® continues to predict college outcomes through students’ 2nd and 3rd years
The relationship between SAT scores and cumulative GPA remains strong and consistent as students progress through their college career.
28
Percent of Students Earning a 2nd Yr and 3rd Yr Cum GPA of a B or Higher by SAT Score Band
Source: Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting Second-Year Grades: 2006 SAT Validity Sample; College Board, 2011; Patterson, B.; Mattern, K.; Kobrin, J.; Validity of the SAT for Predicting Third-Year Grades: 2006 SAT Validity Sample; College Board, 2011;
The SAT® continues to predict college persistence
29
College Retention Rates by SAT Score Band
The SAT ® provides incremental validity over HSGPA in predicting college retention
Once again we see that the SAT in combination with High School grades provides a greater understanding of how students might perform
31
Incremental Validity of SAT Scores over HSGPA for Predicting Second Year Retention
Source: Patterson, B.; Mattern, K.; The Relationship between SAT Scores and Retention to the Second Year: 2007 SAT Validity Sample; College Board, 2011
The SAT ® helps predict four-year graduation rates
32
Rates by score band with continuous enrollment at original institution
The SAT ® provides incremental validity over HSGPA in predicting four-year graduation rates
33
Four-Year Graduation Rates by SAT Scores, holding HSGPA constant
External validity studies confirm the value and validity of the SAT ®
34Sources: University of Minnesota - (Sackett, Kuncel, Arneson, Waters, and Cooper, 2009); University of Georgia (Cornwell, Mustard, and Parys, 2008); University of
California (Agronow, 2007)
SAT ® Fairness
35
SAT ® score performance does vary by sub-group
The existence of score gaps amongst different groups does not necessarily indicate that the assessment is unfair or biased.
36Source: 2011 College-Bound Seniors
The achievement gap exists among many measures of academic achievement and attainment
37
Sources: 2011 College-Bound Seniors; U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, High School Transcript Study (HSTS), various years, 1990–2009. National Center for Higher Education Management Systems, 2009; U.S. Census Bureau, 2006 American Community Survey (ACS)
% - Percent of population age 18-24 with a high school credential, 2006
Average CR+M+W
Average GPA
Academic Achievement Academic Attainment
% National six-year graduation rates of bachelor's degree-seeking students, 2007
Unfortunately, inequities in American society and education exist as evidenced by multiple academic measures
TESTAfrican
AmericansAsian
AmericansHispanics Caucasia
ns
SAT Verbal 434 498 456 526
SAT Math 426 526 460 528
ACT Composite 17.1 21.8 18.9 21.7
GRE Verbal 391 487 438 495
GRE Quant 416 598 482 540
GMAT 416 533 492 531
LSAT 142.7 152.7 145.2 153.5
Achievement gaps persist beyond undergraduate studies
– Regular Curriculum Surveys
• Ensure the content on the SAT aligns with what is taught in high school and what colleges expect entering freshman to know and understand
– Test Development Committees
• Comprised of high school and college faculty review all test items and test forms
– Pre-Test (variable section)
• Field tests new items to collect statistics about item performance, ensure fairness for students of all backgrounds and that students of equal ability perform the same
– Rigorous Psychometric Analysis
• Evaluates the performance of each item to validate that items are performing as expected
The SAT ® is rigorously developed and researched to ensure fairness
The SAT® is the most rigorously researched and designed standardized test in the world.
39
Every question is field-tested across the entire testing population •All 50 U.S. states•Over 170 countriesevery question goes through a sensitivity review to avoid concerns with:•Gender•Ethnicity/Race•Disabilities•Controversial topics like war, violence and politics
Gardens in one form or another has been planted, tended, and harvested since antiquity by people from a wide variety of cultures. No error.
Easier/more difficult for which gender or racial/ethnic group(s), matched on total score?
Eliminating Questions that Behave Differently for Groups of Students
40
African Americans – more difficult
Asian Americans – more difficult
The SAT ® holds as a valid predictor of college performance by all sub-groups
41
Correlation* of SAT® and High School GPA to First-Year College GPA by Race/Ethnicity
When we evaluate the predictive validity of the SAT by racial/ethnic sub-groups we see that SAT continues to be a strong and valid predictor of college performance, in most cases even more so than high school GPA
RACE/ETHNICITY SAT (CR+M+W)
HSGPA SAT+HSGPA
American Indian .54 .49 .63
Asian American .48 .47 .56
African American .47 .44 .54
Hispanic .50 .46 .57
White .53 .56 .63
Total .56 .56 .64
The SAT in combination with HSGPA remains the best predictor of college performance across all sub-groups
Extensive external research confirms that the SAT ® is not biased
42
There is a substantial body of literature indicating that individual
item bias has been largely mitigated in today’s admission test due to extensive external research and development of question items
on both the SAT and ACT®.
-NACAC Testing Commission Report-September 2008
Interpreting and Comparing Scores
43
• Each section score is reported on a 200- to 800- point scale, each section is an independently valid predictor of college success.
• The Writing sub-score is a combination of a multiple-choice score from 20-80 (70%) and an essay score from 2 to 12 (30%)
• Each essay is independently graded by two qualified readers
• The SAT is designed so that a student who answers about half of the questions correctly will receive an average score of approximately 500.
• Scores on any standardized assessment are approximations rather than precise measures of skill. The standard error of measure (SEM) of the SAT usually falls in a range of 30 points for reading and mathematics and 40 points for writing above or below a student’s true skill level.
• When comparing scores there must be a difference of 50-60 points before more skill can be assumed in one area than another.
Interpreting SAT Scores
44
Comparing SAT ® and ACT ® scores
45
•The ACT and SAT are scored on different scales
•The concordance table provides a way for individuals and institutions to compare a student’s performance on one exam with their likely performance on a second exam
•Comparing percentiles is not accurate because the pools of students taking the two tests are different
Note: The concordance table is based on scores from over 300k students who took both versions of the tests with writing from the Class of 2006
The population of students taking the SAT® perform better than the population taking the ACT ® across all sub-groups
To compare the performance of students or populations of students you should concord an ACT score to the SAT scale
46
2011 National Composite ACT Scores
Concorded 2011 ACT Scores to
SAT CR+M Scale
2011 National Combined SAT CR+M
Total 21.1 994 1011
American Indian 18.6 894 972
Asian 23.6 1094 1112
Black 17 830 855
Hispanic 18.7 898 914
White 22.4 1046 1063
Sources: 2011 College Bound Seniors; ACT: Condition of College and Career Readiness, 2011
Resources – For Students
47
SAT Practice Tools At a Glance for Students
5
Practicing for the SAT and SAT Subject Tests Planning for Test Day
SAT Question of the Day – FREE
SAT Question of the Day Mobile App – FREE
SAT Practice Questions – FREE
SAT Subject Test Practice Questions – FREE
SAT Practice Test – FREE
Mathematics Review – FREE
Effective Writing Review – FREE
SAT® Skills Insight™ – FREE
The SAT® Practice Booklet – FREE
Official SAT Study Guide™: 2nd Ed.
Official SAT Subject Test Study Guide™: 2nd Ed.
The Official SAT Online Course™
My SAT Study Plan™ – FREE
Answers Imagined – FREE
SAT Test Taking Approaches – FREE
SAT Essay Strategies – FREE
SAT Subject Test Taking
Approaches – FREE
SAT Subject Test Recommended Skills
and Prerequisites – FREE
SAT Subject Test Web Resources – FREEExpanded Practice QuestionsAnswer ExplanationsRecommended Preparation
Important Test Day information on:
How to Do Your Best
What to Bring
Standby Testing
If You’re Absent
SAT Test Center Closing
Make-up Testing
Test Security and Fairness
Helping Students Get Ready
Models, Lesson Plans, and Strategies for:Argumentative writing skillsSchool-based SAT PracticeThe Official SAT Teacher’s Guide™ESL/ELL students
Professional Development Workshops:SAT Skills InsightAnimating Student Writing Holistic Scoring Workshop School-Based SAT PracticeWriting Preparation for Educators of ESL/ELL Students
Most SAT Practice Tools are FREE!
49
SAT Skills Insight
• SAT Skills Insight identifies the academic skills that typical students should focus on to improve their scores, depending on their target score range
• Free, online resource, paired with My SAT Online Score Report to make SAT In Focus
• Sample SAT questions and answers help students better understand the skill descriptions given
SAT® Offers the Most Generous Fee Waiver Program of Any College Entrance Exam
More than 360,000 low-income students in the graduating class of 2011 benefitted from SAT Fee Waivers (1 in 5)
More than $35 million in services made available at no cost to low-income high school students last academic year
Eligible students can receive:
Two SAT fee waivers and two SAT Subject Tests™ fee waivers (up to six Subject Tests)
Four Free Additional Flexible Score Reports
Free Question and Answer Service or Student Answer Service
Four Request for Waiver of College Application Fee forms
Discount on The Official SAT Online Course™
Resources – For Institutions
51
There are many factors that colleges and universities use to assess students today, but the SAT remains a key component, along with high school grades and a rigorous curriculum.
The SAT:– Provides a national yardstick to compare students across the country
• High schools vary widely in courses, teachers and grading practices
• Grade inflation is more common than ever
– Is a strong predictor of college success and retention• As good as high school grades at predicting student performance in the first,
second and even third years of college
• Gives students a better sense, based on their scores, of how ready they are academically for the rigors of college
The Importance of the SAT® for Institutions
52
To assist colleges and universities in conducting research about their students, the College Board offers the Admitted Class Evaluation Service (ACES™), a free online service that predicts how admitted students will likely perform at your institution and how successful they can be in specific classes.
ACES placement validity studies predict how students will perform on different academic measures by comparing student performance in selected courses with student scores on various College Board assessments*.
Admitted Class Evaluation Service™
* SAT ®, SAT Subject Tests™, ACCUPLACER® tests, AP ® Exams and CLEP ® exams
ACES admission validity studies identify which measures best predict a student’s future performance and recommends the best combination of predictors for your institution.
53
