Upload
others
View
8
Download
0
Embed Size (px)
Citation preview
Validation and ACCUPLACER®
ACCUPLACER National ConferenceSan Francisco, CA
June 29, 2012June 29, 2012
Mary-Margaret Kerns, Ph.D.Executive Directorecut e ectoHigher Education ResearchResearch and Development
Structure of presentation
• Validity
• Admitted Class Evaluation Service™ (ACES™)
• ACES ACCUPLACER Placement Validity StudyC S CCU C ace e t a d ty Study
2
Reliability is not enough
• If test scores are used to make inferences about an examinee’s ability, the test must be both y,reliable and valid. • Reliability is a prerequisite for validity and refers to the
ability of a test to measure a particular trait consistently.
• Tests can be highly reliable and still not be valid for a particular purposeparticular purpose.
• Determining the reliability of a test is an important first step but not the defining step in determining the validity p g p g yof a test.
3
What is test validity and test validation?
• A test that is perfectly appropriate and useful in one situation may be inappropriate or insufficientone situation may be inappropriate or insufficient in another.
For example a test valid for determining high school• For example, a test valid for determining high school graduation may be inappropriate for use in college placement.
• Test validity, or the validation of a test, explicitly means validating the use of a test in a specific g pcontext, e.g., placement into a course.• Therefore, it is important to study the test results in the , p y
setting in which they are used.4
Validity Evidence 5
Validity refers to the degree to which evidence and theory support the interpretations of test scores entailedtheory support the interpretations of test scores entailed by proposed uses of tests.
Standards of Educational and Psychological Testing
(AERA, APA, NCME, 1999)( , , , )
5
Validity Evidence
• Content Validity
C t t V lidit• Construct Validity
• Consequential Validity
• Criterion-Related Validity
6
Criterion-Related Validity
• Criterion-related validity looks at the relationship between a predictor and a criterion / outcome.
• A criterion-related validity study is completed by collecting the test score (predictor) information and information on grades (criterion / outcome) for the same students.
• E.g., ACCUPLACER Reading Comprehension scores and ENG100 grades
• E.g., ACCUPLACER Elementary Algebra scores and MATH100 grades
• Refers to the “power” or usefulness of test scores to predict future performance.
• E.g., future performance as academic success (or non-success) in college
7
ACCUPLACER TESTS COURSE GRADES
Arithmetic Elementary Reading Sentence WritePlacer ENG100 Math100
Algebra Comprehension Skills Grade
Student A 85 70 35 88 6 A B
Student B 92 100 108 99 7 C A
St de t C 65 52 24 30 1 D CStudent C 65 52 24 30 1 D C
Student D 90 73 65 68 3 C B
Student E 79 80 58 75 6 A B
Student F 90 95 87 99 8 A A
Student G 69 98 42 53 1 C C
Student H 85 102 91 95 4 C B
Student I 99 108 118 115 8 A A
Student J 75 68 115 100 7 B BStudent J 75 68 115 100 7 B B
8
E
F
I
ACCUPLACER READING COMPREHENSION SCORES and ENG100 GRADES
A
J
H
D
RAD
ES
B
CG
ENG
100
GR C
D
DH
B
C
20 30 40 50 60 70 80 90 100 110 120
READING COMPREHENSION SCORES
F
READING COMPREHENSION SCORES
9
10
11
12
13
E
F
I
ACCUPLACER Reading Comprehension Scores and ENG100 Grades
90
J
H
D
Succ
ess
75
H
G
Prob
abili
ty o
f S 60
45
DH
B
C
20 30 40 50 60 70 80 90 100 110 120
30
ACCUPLACER Reading Comprehension Scores
14
http://professionals.collegeboard.com/higher-ed/validity/aces
15
Admitted Class Evaluation Service (ACES)
• ACES offers two models of validity studies: • Admission
• PredictivePredictive• Placement
P di ti• Predictive
• Concurrent
16
Placement Validity Studies
• ACES is able to process predictive placement validity studies using scores from:validity studies using scores from: • SAT
• SAT Subject Tests
• ACCUPLACER Tests
17
Number of Predictors
• In some situations, one factor can predict a certain outcomecertain outcome.
• The more types of relevant information, the better you are able to predict the criterionyou are able to predict the criterion.
18
Designing a Predictive Validity Study
• You are predicting success on the criterion (end of course grade) using a predictor or(end-of-course grade) using a predictor, or predictors
The predictor information is gathered at some• The predictor information is gathered at some point before the student enters the course
It is important that the predictor is current and• It is important that the predictor is current and that little or no instructional intervention has occurred between the predictor and the start of pthe course
19
Designing a Predictive Validity Study
• The predictor/s and criterion need to be collected from all students at the same point in the coursefrom all students at the same point in the course sequence.
• In an effort to increase sample size some investigators• In an effort to increase sample size, some investigators include students who took the predictor, were then placed in a developmental course, and then take the college-level courselevel course.
• The intervention of the developmental course changes the relationship between the predictor and the college-level p p gcourse criterion, making the data incompatible.
20
Designing an ACES Predictive Validity Studyy
• Number of students in the study• A reasonably large sample is important to increase the
likelihood of a sample being similar to the population it is intended to representp
• Validity coefficients are extremely sensitive to chance fluctuations (a change in one score can make a large difference))
• The minimum number of student required for an ACES study depends on the number of predictors used
• 1 to 3 predictors requires a minimum of 30 students
• 4 predictors, a minimum of 40 students
5 predictors a minimum of 50 students• 5 predictors, a minimum of 50 students
21
Designing a Predictive Validity Study
• In order to effectively evaluate how well a predictor is working, ACES requires a range of at least 4 points onworking, ACES requires a range of at least 4 points on a scale for the criterion.
• Students with pass/fail grades should not be included in p gan ACES study.
22
Predictive Validity Study
• Statistical factors play a role in predictive validity t di (i d th it d f di tistudies (i.e. reduce the magnitude of predictive
validity coefficients)• Small sample sizes
• Measurement unreliability
• Restriction of range
23
http://professionals.collegeboard.com/higher-ed/validity/aces
24
Requesting a Placement Study
• You may analyze up to five separate courses in your Placement Validity Study.y y y
• For each course you will begin by specifying the tests and the predictors you are most interested i l tiin evaluating. • You may use a single predictor, or up to five predictors,
for a single course.for a single course.
25
26
ACES Data Submission
• Choose from a variety of common formats including:g• Excel • Access • SPSS • SAS • ASCII delimited format
• Upload the file directly from your PC to ACES p y y• ACES automatically encrypts the data during
transmission to protect confidentiality
27
Getting an ACES Study
• ACES notifies the institution by e-mail when the study is completedstudy is completed
• The document is password protected for confidentiality and is encrypted until downloaded y ypat the institution
• Results of ACES studies are confidential and only l d t th i di id l th t t d th t dreleased to the individual that requested the study
• Placement studies take up to 40 business days to process from time of successful data submissionprocess from time of successful data submission
28
ACES Validity Handbook(http://professionals.collegeboard.com/higher-ed/validity/aces/handbook)
• It is designed to serve as a general reference for validity and includes information about validity beyond what is specifically applicable to ACESis specifically applicable to ACES.
• It includes specific information about• It includes specific information about the types of validity studies and their design that are available through the ACES system and helps interpret y p pACES study results.
29
30
ACES Placement Validity Study
• The predictive validity study provides two probability tables for cut scores:probability tables for cut scores:• For a course grade of B or higher
F d f C hi h• For a course grade of C or higher
31
Section 1: The purpose of this report
• When requesting this report, you indicated that you wish to study placement in two courses.
• You indicated you plan to use scores from ACCUPLACER tests to predict success in English 100.
The specific predictor variables are: Reading Comprehension• The specific predictor variables are: Reading Comprehension and Sentence Skills.
• You indicated you plan to use scores from ACCUPLACER tests to predict success in Math 100.
• The specific predictor variables are: Elementary Algebra andCollege Level Math.g
32
Section 2: Your sample of students
33
Section 2: Your sample of students
34
Section 3: Strength of Prediction
35
36
ACCUPLACER Validity Presentation ResourcesResources
Admitted Class Evaluation Service (ACES™) http://professionals.collegeboard.com/higher-ed/validity/aces
ACES Placement Validity Study http://professionals.collegeboard.com/higher-ed/validity/aces/placement
Placement Validity Report for ACCUPLACER Sample// f / f /http://professionals.collegeboard.com/profdownload/aces-sample-placement-validity-report-
accuplacer.pdf
Predictive Validity of ACCUPLACER Scores for Course Placement: A M t A l iA Meta-Analysishttp://professionals.collegeboard.com/data-reports-research/cb/validity-accu-placement
37
Questions, Comments, Suggestions
• Researchers are encouraged to freely express their professional judgment. Therefore, points oftheir professional judgment. Therefore, points of view or opinions stated in College Board presentations do not necessarily represent official C ll B d iti liCollege Board position or policy.
• Please forward any questions, comments, and suggestions to:
mmkerns@collegeboard [email protected]
38