Embed Size (px)
Citation preview
MELMACPeer Learning
Understanding the Redesigned SAT®, its Impact
in Maine, and Validity
March 7, 2017
• SAT Suite of Assessments
• Impact of SAT Suite of Assessments
• Preliminary Validity StudyAgenda
The SAT Suite of Assessments
Readiness
Baseline
Check-in
and Focus
Connect to
College
200-800 points
About the Redesigned SAT®
200-800 points
Math Essay
(separate scores)
3 Hours 50 Minutes
Evidence-Based
Reading and Writing
200-800 points
4
Scores and Score Structures
SAT® Essay Scores
The SAT Essay reports scores on three dimensions:
Reading2–8 Scale
Analysis2–8 Scale
Writing2–8 Scale
6
Longitudinal Progress Monitoring
7
► Section Scores will be placed on a vertical scale.
This same concept will hold true for the Test and Cross-
Test Scores as well as Total Score.
Vertical Score Scale
Total scores, section scores, test scores, and cross-test scoresare vertically scaled.
► Improved scores demonstrate growth from assessment to assessment.
► Vertical scaling allows educators to monitor growth across grades.
Subscores are not vertically scaled.
► Subscores provide information allowing students to pinpoint areas for improvement.
► Improved scores do not demonstrate growth from assessment to assessmentat the subscore level.
8
Scores Across the SAT®
Suite of Assessments
The redesigned SAT will be the anchor of a vertically aligned SAT Suite of Assessments.
ERW 480MATH 530
► Working together, the tests in the SAT® Suite of Assessments provide college and career readiness benchmarks and consistent feedbackfor measuring student progress.
► Benchmarks indicating college and career readiness are determined usingSAT Suite data and first-year college performance data.
► The benchmark indicates that students who earn that score or higher have a 75% likelihood of earning a C or better in a first semester, credit-bearing course in the same subject area.
► The PSATTM 8/9 will provide benchmarks and norms for both 8th and 9th grades.
► PSATTM 10 will provide benchmarks and norms for 10th grade.
► The PSAT/NMSQT® will provide benchmarks and norms for 10th-and 11th-graders.
10
College and Career Readiness Benchmarks
11
Set at section level.
Grade-level benchmarks are based on expected student growth toward the SAT Benchmarks:
Grade 8 Grade 9 Grade 10 Grade 11 SAT
Section Level390 410 430 460 480 ERW
430 450 480 510 530 MATH
12
(Pre-Redesigned) SAT Benchmarks SAT Benchmarks
Based on data obtained from four-year institutions Based on data from two- and four-year institutions
Based on an average of course outcomes (GPA) Based on course outcomes from related content areas
Includes coursework in first year Includes coursework in first semester
All coursework included Coursework is related to the relevant SAT section
All coursework included Only non-remedial coursework included
All coursework included Only introductory (100-level) coursework included
The percent college ready is based on achieving the composite
score
The percent college ready is based on achieving both section
benchmarks (Evidence-Based Reading and Writing and Math)
Calculated based on a 65% probability of earning at least a 2.67 (B-
) in the first year of college
Calculated separately based on a 75% probability of earning at least
a C in related first semester courses
• Assess only what students are already learning in the classroom, and onlythose things students will use over and over again to be successful.
• Ensure all students claim the opportunities they've earned.
• Assessments must earn students and their families something worthwhile.
• The only things that change performance are productive practice and great courses.
Assessment without Opportunity is DOA: Four Rules for Assessment
Student Report (Paper/PDF Version)
14
• The SAT Suite of Assessments creates a multi-
year trajectory, allowing students to see how
they’re progressing throughout high school
• “SAT vocabulary words” are gone
• No penalty for guessing
• Focused on math that matters most
• Reflects what students learn in class
• Free test practice for all
• 2.5 million students are using Official SAT
Practice with Khan Academy®
• At roughly equal rates across all income
levels, races, and ethnicities
Making Excellence Easier: The New SAT
Steps to Link College Boardand Khan Academy Accounts
Step 1Log in or create a Khan Academy account at
satpractice.org
Step 2When prompted, students can agree to link their Khan
Academy and College Board accounts.
Step 3Students sign in or create a College Board account
Step 4When prompted, students click “Send” to authorize the
account linking
Step 5Students start practicing on Official SAT Practice on
Khan Academy!
• 12,065 Maine Juniors (3rd year high school students) took the SAT in April (91% of ME public school students)
• 1,800+ more students have SAT scores in the graduating class of 2017
• Maine Dept. of Ed. opted to administer the SAT with Essay to all of its students
• Thousands of students and families were connected to opportunities as part of the Maine DOE state assessment system; above and beyond the school accountability results
Maine’s 2016 SAT School Day Administration Overview
ConcordanceTables on
Three Levels
• Total to Total
• 1600 to 1600
• 2400 to 1600
• Section to Section
• CR+W to ERW
• Math to Math
• Section to Test (or Test to Section)
• Critical Reading to Reading Test
• Writing to Writing and Language
Test
In addition to the tables themselves, threetools will be available
Smartphone Apps
iPhone and Android apps — provide
the same functionality as the old score
conversion slider.
Web
Provides the same functionality as
the apps, enabling users to make
score-to-score conversions.
Large-Scale Concordance
Higher education institutions may also
need tools to convert large numbers of
scores quickly.
Redesigned SAT Pilot Predictive Validity Study
• As part of the redesign of the SAT, the College Board conducted a pilot study to examine the predictive validity of redesigned SAT scores with college outcomes
• Fifteen four-year institutions administered a pilot form of the redesigned SAT to a sample of first-year, first-time college students very early in the fall semester of 2014
• The participating institutions varied by size, selectivity, public/private, and region
• 2,050 students were included in the final sample, and varied by race/ethnicity, best language, parental education, and parental income level
21
Key Findings • The redesigned SAT covers the content and skills that research shows matter most for college readiness – and it remains as predictive of college success as the old SAT
• Redesigned SAT scores improve the ability to predict college performance above high school GPA alone
• There is a strong, positive relationship between redesigned SAT section scores and grades in matching college course domains, suggesting that the redesigned SAT is sensitive to instruction in English language arts, math, science, and history/social studies
Interpreting Correlation Coefficients
Small correlation has an absolute value of approximately 0.1
Medium correlation has an absolute value of approximately 0.3
Large correlation has an absolute value of
approximately 0.5 or higher
Cohen, J. (1988). Statistical power analysis for behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Mean FYGPA by SAT Total Score Band
2.682.89
3.143.33
3.58
1.00
1.50
2.00
2.50
3.00
3.50
4.00
600-790(n=41)
800-990(n=219)
1000-1190(n=435)
1200-1390(n=764)
1400-1600(n=589)
FYG
PA
SAT Total Score Band
(graphical depiction of the multiple correlation of SAT with FYGPA of 0.53)
Note: Results based on fewer than 15 cases are not reported (e.g., score band 400–590, n = 2).
25
(graphical depiction of 0.10 incremental validity added by SAT above HSGPA to predict FYGPA)
Incremental Validity of the SAT
2.81 2.783.032.95
3.213.03
3.363.313.59
1.00
1.50
2.00
2.50
3.00
3.50
4.00
B A
FYG
PA
HSGPA
600-790 800-990 1000-1190 1200-1390 1400-1600
Note: HSGPA ranges are defined as follows: “A” range: 4.33 (A+), 4.00 (A), and 3.67 (A-); “B” range: 3.33 (B+), 3.00 (B), and 2.67 (B-). Results based on fewer than 15 cases are not reported. Not reported are all of the C or lower HSGPA group, HSGPA B: 400–590 SAT score band, and HSGPA A: 400–590 and 600–790 SAT score bands.
Relationship Between SATEvidence-Based Reading and Writing Scores and Course Grades in the Same Domain
Note: Results based on fewer than 15 cases are not reported (e.g., score band 200–290, n = 1). Average English course grade includes
first-semester courses that are reading and writing intensive (excluding foreign and classical languages).
2.64
2.89
3.13
3.43
3.65
1.00
1.50
2.00
2.50
3.00
3.50
4.00
300-390(n=31)
400-490(n=163)
500-590(n=321)
600-690(n=505)
700-800(n=363)A
ve
rag
e C
ou
rse
Gra
de
in
En
gli
sh
SAT EBRW Section Score Band
Relationship Between SAT Math Section Scoresand Course Grades in the Same Domain
27
2.09
2.502.59
2.95
3.27
1.00
1.50
2.00
2.50
3.00
3.50
4.00
300-390(n=29)
400-490(n=71)
500-590(n=175)
600-690(n=368)
700-800(n=429)
Ave
rag
e C
ou
rse
Gra
de
in
M
ath
em
ati
cs
SAT Math Section Score Band
Note: Results based on fewer than 15 cases are not reported (e.g., score band 200–290, n = 1). Average math course grade
includes first-semester course work in algebra, precalculus, calculus, and statistics.
Relationship Between Analysis in Science Cross-Test Scores and Course Grades in the Same Domain
2.48
2.702.83
3.12
3.43
1.00
1.50
2.00
2.50
3.00
3.50
4.00
15-19(n=26)
20-24(n=139)
25-29(n=276)
30-34(n=543)
35-40(n=402)
Ave
rage
Co
urs
e G
rad
e in
Sci
en
ce
SAT® Analysis in Science Cross-Test Score BandNote: Results based on fewer than 15 cases are not reported (e.g., score band 10–14, n = 0). Average science course grade includes first-semester course work in natural sciences, health sciences, and engineering.
Relationship Between Analysis in History/Social StudiesCross-Test Scores and Course Grades in the Same Domain
2.54
2.983.14
3.443.62
1.00
1.50
2.00
2.50
3.00
3.50
4.00
15-19(n=38)
20-24(n=125)
25-29(n=313)
30-34(n=401)
35-40(n=372)
Ave
rage
Co
urs
e G
rad
e in
His
tory
/So
cial
Stu
die
s
SAT® Analysis in History/Social Studies Cross-Test Score BandNote: Results based on fewer than 15 cases are not reported (e.g., score band 10–14, n = 3). Average history/social studies course grade includes first-semester course work in history and social sciences.
• Retention information was available for 1,991 of the original 2,050 students from the National Student Clearinghouse.
• 1,879 students were retained to the second year
• 112 students were not retained to the second year
• 59 students were missing retention information
• The following figures describe the relationship between redesigned SAT scores and retention to the second year for the 1,991 students with retention information.
30
Retention to the Second Year
89%86%
88%92% 93%91%
97%93%
97%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
B A
Pe
rce
nt
of
Stu
de
nts
Re
tain
ed
to
th
e
Se
co
nd
Ye
ar
HSGPA
600-790 800-990 1000-1190 1200-1390 1400-1600Incremental Validity of the SAT: Retention to the Second Yearby SAT Total Score Band, Controlling for HSGPA
Note: Results based on fewer than 15 cases are not reported. Sample sizes within each band
include both retained and non-retained students. For students with a HSGPA of A and an SAT
score of 1400-1600, the average second year retention rate was 0.9749 which rounds to 97%.
National Validity Study of the Redesigned SAT
• The College Board will launch a more comprehensive, longitudinal national SAT validity study in partnership with colleges and universities
• This study will examine the relationship between SAT scores and important college outcomes such as GPA, course grades, persistence, and completion
• Institutions interested in participating in the upcoming national validity study of the redesigned SAT should contact [email protected] prior to September 2018
• Initial findings from this large-scale study will be available in 2019
Questions? Pat Doyle
Director State and District Partnerships
