Using CogATUsing CogAT
David LohmanDavid LohmanInstitute for Research and Policy on Institute for Research and Policy on
AccelerationAccelerationBelin-Blank CenterBelin-Blank Center
&&Iowa Testing ProgramsIowa Testing Programs
University of IowaUniversity of Iowa
http://faculty.education.uiowa.edu/dlohman/
TopicsTopics Distinguishing between ability & Distinguishing between ability &
achievementachievement Overview of CogATOverview of CogAT Comparing CogAT with other ability testsComparing CogAT with other ability tests Interpreting CogAT scoresInterpreting CogAT scores General issues in selectionGeneral issues in selection Identification of talent in special Identification of talent in special
populationspopulations Combining Achievement, Ability, & Teacher Combining Achievement, Ability, & Teacher
ratings: the Lohman – Renzulli matrixratings: the Lohman – Renzulli matrix
Distinguishing Distinguishing between ability and between ability and
achievementachievement
Puzzlements for common Puzzlements for common interpretations of ability & interpretations of ability &
achievementachievement Is ability more biologically based?Is ability more biologically based?
Most studies show same heritability for IQ (Gf) Most studies show same heritability for IQ (Gf) and achievement tests (Gc)and achievement tests (Gc)
Lower relative achievement than ability = Lower relative achievement than ability = underachievementunderachievement But there are an equal number of “overachievers”But there are an equal number of “overachievers”
Status scores (IQ, PR) show good stabilityStatus scores (IQ, PR) show good stability But one must keep getting better to retain that IQBut one must keep getting better to retain that IQ Between 9 – 17 r(True IQ) = .75. Between 9 – 17 r(True IQ) = .75.
60% in top 3% at 9 NOT in top 3% at age 1760% in top 3% at 9 NOT in top 3% at age 17
Fluid abilities invested in experience Fluid abilities invested in experience to produce particular constellations to produce particular constellations of crystallized abilities?of crystallized abilities? Only for very young childrenOnly for very young children Thereafter, crystallized abilities -> fluidThereafter, crystallized abilities -> fluid
Level 1. NominalismLevel 1. Nominalism (Most (Most people here)people here)
““ability” and “achievement” are ability” and “achievement” are separate (Jangle fallacy –T. separate (Jangle fallacy –T.
Kelley, 1927)Kelley, 1927)
AbilityAbility AchievementAchievement
Ability Achievement
Level 2. Oh, Oh – there’s more Level 2. Oh, Oh – there’s more overlap than uniqueness here!overlap than uniqueness here!
Its all ‘g’ (any indicant will do)Its all ‘g’ (any indicant will do) Its all just a product of experienceIts all just a product of experience Preserve stage 1 beliefs –Preserve stage 1 beliefs –
Purge ability of visible achievement (e.g. Purge ability of visible achievement (e.g. measure “process” or use only “nonverbal” measure “process” or use only “nonverbal” measures)measures)
Level 3. Island kingdoms –Things Level 3. Island kingdoms –Things
get even more complicatedget even more complicated ((most most scholars of human abilities)scholars of human abilities) Effects of language, culture, and Effects of language, culture, and
experience on the development of experience on the development of ability (“All abilities are developed” ability (“All abilities are developed” Anastasi)Anastasi)
Experience alters the structure of Experience alters the structure of the brainthe brain
Mental processes do not exist Mental processes do not exist independently of knowledge. independently of knowledge.
70
75
80
85
90
95
100
105
1910 1920 1930 1940 1950 1960 1970 1980 1990 2000
Year
IQ o
n t
he
19
95
Sc
ale
Gains in Wechsler-Binet IQ for the U.S. White population. Sources J. Horgan (1995) and D. Schildlovsky.
Example of Flynn EffectExample of Flynn Effect
Proportion of variance in WISC Full Scale IQ at age 7 accounted for by genetic factors as a function of socioeconomic status (SES)
Turkheimer et al. (2003) Psychological Science, 14 (6). N= 319 twin pairs.43% White, 54% Black. Most families poor.
Low High
Fluid-Crystallized Continuum (1)Fluid-Crystallized Continuum (1)
Fluid Crystallized
Physical skills
General physical fitness
Basketball
Swimming
Football
Field hockey
Volleyball
Wrestling Cycling
Fluid-Crystallized Continuum (1)Fluid-Crystallized Continuum (1)
Fluid Crystallized
Cognitive abilities
Physical skills
General fluid ability (Gf)
Science achievement
Math achievement
Social studies achievement
Knowledge of literature
Specific factual
knowledge
General physical fitness
Basketball
Swimming
Football
Field hockey
Volleyball
Wrestling Cycling
Fluid-Crystallized Continuum (2)Fluid-Crystallized Continuum (2)
Fluid Crystallized
Cognitive abilities
Physical skills
General fluid ability (Gf)
Science achievement
Math achievement
Social studies achievement
Knowledge of literature
Specific factual
knowledge
General physical fitness
Basketball
Swimming
Football
Field hockey
Volleyball
Wrestling Cycling
A common ability-achievement A common ability-achievement spacespace
Level 4. Systems theoriesLevel 4. Systems theories (A (A handful)handful)
Aptitude Theory (Richard Snow)Aptitude Theory (Richard Snow)
Sidesteps the issue of defining Sidesteps the issue of defining intelligence;intelligence;
starts with expertise & the starts with expertise & the contexts in which it is developed & contexts in which it is developed & displayed, displayed,
readiness to learn in those readiness to learn in those contextscontexts
Overview of Overview of CogATCogAT
Some HistorySome History
Lorge -Thorndike Intelligence testLorge -Thorndike Intelligence test Cognitive AbilitieCognitive Abilitiess Test Test
Form 1 1974Form 1 1974 Forms 2 – 3 (no Composite score)Forms 2 – 3 (no Composite score) Forms 4 – Thorndike & Hagen – Comp Forms 4 – Thorndike & Hagen – Comp
scorescore Form 5 – HagenForm 5 – Hagen Form 6 – Lohman & HagenForm 6 – Lohman & Hagen
Co-normed with the ITBS & ITEDCo-normed with the ITBS & ITED
Primary uses of CogATPrimary uses of CogAT
To guide efforts to adapt instruction To guide efforts to adapt instruction to the needs and abilities of studentsto the needs and abilities of students
To provide an alternative measure of To provide an alternative measure of cognitive developmentcognitive development
To identify students whose predicted To identify students whose predicted levels of achievement differ levels of achievement differ markedly from their observed levels markedly from their observed levels of achievementof achievement
Primary Battery (K-2)Primary Battery (K-2)
Oral VocabularyVerbal Reasoning
Verbal Reasoning.....
Relational ConceptsQuantitative Concepts
Quantitative Reasoning.....
Figure ClassificationMatrices
Nonverbal Reasoning.....
General Reasoning Ability
No readingTests untimed (paced by teacher)Mark directly in booklet
Multilevel Battery (gr. 3-Multilevel Battery (gr. 3-12)12)
Verbal ClassificationSentence Completion
Verbal Analogies
Verbal Reasoning......
Quantitative RelationsNumber Series
Equation Building
Quantitative Reasoning.....
Figure ClassificationFigure AnalogiesFigure Analysis
Nonverbal Reasoning.....
General Reasoning Ability
Tests timedSeparate Answer sheetCommon Directions
3 Separate Test 3 Separate Test BatteriesBatteries
(Not one)(Not one)
ScoresScores
Raw score = number correctRaw score = number correct Scale score – USSScale score – USS
Within level - map number correct on to Within level - map number correct on to a scale whose intervals are a scale whose intervals are approximately the same sizeapproximately the same size
Between levels – maps number correct Between levels – maps number correct on different levels of the test on to a on different levels of the test on to a single, common, developmental scalesingle, common, developmental scale
A
USS Scale
etc
B
C
D
Relationships among Stanines, Percentile Relationships among Stanines, Percentile Ranks, and Standard Age ScoresRanks, and Standard Age Scores
134 - 150
CompositesComposites
Composite scoresComposite scores Partial VQ, VN, QNPartial VQ, VN, QN Full – VQN or C [do NOT use for Full – VQN or C [do NOT use for
screening]screening] Primary BatteryPrimary Battery
V or (VQ) versus NV or (VQ) versus N Multilevel BatteryMultilevel Battery
V versus QNV versus QN
Consequential Validity:Consequential Validity:Score warnings Score warnings
Age out of rangeAge out of range Age unusual for coded gradeAge unusual for coded grade Estimated test levelEstimated test level Level unusual for coded gradeLevel unusual for coded grade Targeted scoreTargeted score Too few items attempted to scoreToo few items attempted to score Many items omitted (slow and Many items omitted (slow and
accurate)accurate) Extremely variable responsesExtremely variable responses
Personal Confidence Personal Confidence IntervalsIntervals
Pattern of item responses aberrant?Pattern of item responses aberrant? Inconsistent across subtests within a battery?Inconsistent across subtests within a battery? Personal Standard Error of Measurement Personal Standard Error of Measurement
(PSEM)(PSEM)
1 25 50 75 99
V 120 89Q 116 84N 125 94
SAS PR
Score ProfilesScore Profiles
CogAT 6 ‘ABC’ Profile CogAT 6 ‘ABC’ Profile systemsystem
Measuring the patternMeasuring the pattern ““A” profiles:A” profiles: Confidence bands overlap for Confidence bands overlap for
all three scores. Scores are at roughly the all three scores. Scores are at roughly the ssAAmeme level level
““B” profiles:B” profiles: One score is One score is aaBBoveove or or BBelowelow the other two scores, which do not differthe other two scores, which do not differ
““C” profiles:C” profiles: Two scores Two scores CContrastontrast
““E” profiles:E” profiles: Extreme B or C profiles Extreme B or C profiles (>=24)(>=24)
““A” ProfileA” Profile
1 25 50 75 99
V 120 89Q 116 84N 125 94
SAS PR
““B” ProfilesB” Profiles
1 25 50 75 99
V 120 89Q 116 84N 100 50
SAS PR
1 25 50 75 99
V 95 38Q 92 31N 110 73
SAS PR
N-
N+
““C” ProfileC” Profile
1 25 50 75 99
V 120 89Q 110 73N 100 50
SAS PR
V+ N-
Extreme “C” ProfileExtreme “C” Profile
1 25 50 75 99
V 120 89Q 107 67N 92 31
SAS PR
SAS Max – SAS Min = 28 E (V+ N-)
Profile LevelProfile Level
Median (middle) age stanineMedian (middle) age stanine
66 A A
55 B (V+) B (V+)
88 C (Q+ V-) C (Q+ V-)
22 E (N+ V-)E (N+ V-)
CogAT6 Profile frequencies for CogAT6 Profile frequencies for students students
in K-12 populationin K-12 population
ProfileProfile
Percent Percent in K-12 in K-12 populatipopulati
onon
AA 3333
BB 4242
B+B+ ( 21)( 21)
B -B - (22)(22)
EE 77
B+B+ (4)(4)
B -B - (3)(3)
CogAT6 Profile frequencies for CogAT6 Profile frequencies for students in K-12 pop. and for students students in K-12 pop. and for students
with two stanine scores of 9with two stanine scores of 9
ProfileProfile
Percent Percent in K-12 in K-12 populatipopulati
onon
Percent Percent in in
Stanine=Stanine=9 group9 group
AA 3333 3737
BB 4242 2727
B+B+ ( 21)( 21) ( 6)( 6)
B -B - (22)(22)
(( 21)21)
EE 77 1919
B+B+ 44 ( 3)( 3)
B -B - 33 ( ( 16)16)
37%
Comparing Comparing CogAT with other CogAT with other
teststests
ReliabilityReliability Many estimates for a given testMany estimates for a given test
Sources of errorSources of error Correlation versus standard error of Correlation versus standard error of
measurement (SEM)measurement (SEM) Correlations depend on sample variabilityCorrelations depend on sample variability Easily misinterpretedEasily misinterpreted
SEMSEM Typical SD of distribution of test scores if Typical SD of distribution of test scores if
each student could be tested many timeseach student could be tested many times
Person-level estimate – Only on CogATPerson-level estimate – Only on CogAT
SEM for SAS scoresSEM for SAS scores
SEM for SAS scoresSEM for SAS scores
SEM for SAS scoresSEM for SAS scores
SEM for SAS scoresSEM for SAS scores
Standard Errors of Standard Errors of Measurement for Individual Measurement for Individual
& Group Tests& Group Tests
WISCWISC-IV-IV SB-VSB-V
CogAT CogAT 66
OLSAT-OLSAT-88 InviewInview RavenRaven NNATNNAT
VerbalVerbal 3.93.9 3.63.6 3.43.4 5.75.7 5.35.3
NonverbalNonverbal 4.24.2 3.93.9 3.73.7 5.85.8 4.54.5 3.03.0 6.16.1
QuantitativeQuantitative 4.54.5 5.35.3 3.33.3
Comp/Full ScaleComp/Full Scale 2.82.8 2.82.8 2.22.2 5.75.7 3.53.5
Standard Errors of Standard Errors of Measurement for Individual Measurement for Individual
& Group Tests& Group Tests
WISCWISC-IV-IV SB-VSB-V
CogAT CogAT 66
OLSAT-OLSAT-88 InviewInview RavenRaven NNATNNAT
VerbalVerbal 3.93.9 3.63.6 3.43.4 5.75.7 5.35.3
NonverbalNonverbal 4.24.2 3.93.9 3.73.7 5.85.8 4.54.5 3.03.0 6.16.1
QuantitativeQuantitative 4.54.5 5.35.3 3.33.3
Comp/Full ScaleComp/Full Scale 2.82.8 2.82.8 2.22.2 5.75.7 3.53.5
Individually-administered tests:Individually-administered tests: SB-V SB-V WISC-IVWISC-IV
Group-administered tests:Group-administered tests: Inview Inview Otis-LenonOtis-Lenon NNATNNAT
CogAT is more reliable CogAT is more reliable thanthan
Conditional Standard Error Conditional Standard Error of Meas.of Meas.
Cogat 6 Verbal Battery: Level A
0.00
5.00
10.00
15.00
20.00
0 5 10 15 20 25 30 35 40 45 50 55 60 65
Number Correct
SE
M USS Score
Raw Score
Conditional SEM's for CogAT6 Verbal USS scores, by test level
Verbal USS K 1 2 A B C D . . .. . . . . . .. . . . . . .
191-195 11.5 9.9 6.5 5.3 4.8 4.3196-200 15.9 11.4 7.5 5.9 5.2 4.5201-205 12.0 12.5 8.5 7.0 5.4 4.8206-210 10.5 7.4 5.9 5.2211-215 17.0 12.5 13.0 13.1 8.9 6.9 5.6216-220 13.4 10.4 8.4 6.2221-225 13.0 14.8 13.2 10.9 7.4226-230 13.9 8.5231-235 14.5 14.8 13.3 10.8236-240 15.0 13.3241-245 16.9 14.3246-250251-255 16.5 14.8256-260 95th PR261-265 15.4266-270 99th PR 16.4
Out of level testing?Out of level testing?
SAS or PR scores?SAS or PR scores? Primary Battery – Multilevel Primary Battery – Multilevel
Battery?Battery? Requires individual testingRequires individual testing Assumes child can use machine-Assumes child can use machine-
readable answer sheetreadable answer sheet Quant battery assumes familiarity with Quant battery assumes familiarity with
numerical operationsnumerical operations Level A – H?Level A – H?
Common time limits & directionsCommon time limits & directions
ValidityValidity Construct Construct
Representation --- all three aspects of Representation --- all three aspects of fluid reasoning abilityfluid reasoning ability
Predictive Predictive Excellent for predicting current and Excellent for predicting current and
future academic achievementfuture academic achievement Predictions the same for all ethnic Predictions the same for all ethnic
groups groups ConsequentialConsequential
No other test comes closeNo other test comes close
Validity:Validity:Construct Construct
RepresentationRepresentation
Carroll’s Three-Stratum Theory of Human Abilities
Gf Fluid Reasoning Abilities
Carroll’s Three-Stratum Theory of Human Abilities
VerbalSequentialReasoning
QuantitativeReasoning
Figural-Inferential Reasoning
Correlation between WISC Full Scale Score and CogAT Composite = .79
Predictive ValidityPredictive Validity Correlations with current and subsequent Correlations with current and subsequent
achievementachievement Within Battery predictions strongWithin Battery predictions strong
Verbal with Reading, Soc Studies (r =.4 - .8)Verbal with Reading, Soc Studies (r =.4 - .8) Quant with Mathematics (r = .4 - .8)Quant with Mathematics (r = .4 - .8) Figural–Nonverbal with Math (r = .4 - .7)Figural–Nonverbal with Math (r = .4 - .7)
Negative for verbal ach. after controlling gNegative for verbal ach. after controlling g Across batteriesAcross batteries
Multiple correlations – typically R = .8Multiple correlations – typically R = .8 Often better than prior achievement in the domainOften better than prior achievement in the domain V and QN partial composite especially usefulV and QN partial composite especially useful
Within ethnic-group correlations the sameWithin ethnic-group correlations the same Implications for TALENT identificationImplications for TALENT identification
Consequential Validity: Consequential Validity: Advice on score Advice on score interpretation?interpretation?
Early 20Early 20thth century theory of ‘culture-fair century theory of ‘culture-fair measure of g’ measure of g’
2121stst century theory of reasoning abilities century theory of reasoning abilities Evidence from research on human abilitiesEvidence from research on human abilities Evidence from predictions of academic Evidence from predictions of academic
achievementachievement Evidence from ATI researchEvidence from ATI research Evidence from cognitive psychologyEvidence from cognitive psychology
Consequential Validity: Consequential Validity: Score useScore use
Does Does everyevery child (teacher) receive child (teacher) receive potentially useful information?potentially useful information?
Specific suggestions for how to use the level Specific suggestions for how to use the level and profile of scores to and profile of scores to Assist the child in learning by adapting Assist the child in learning by adapting
instruction better to meet his/her learning styleinstruction better to meet his/her learning style Build on cognitive strengthsBuild on cognitive strengths Shore up weaknessesShore up weaknesses
Interpretive Guide for Teachers & Interpretive Guide for Teachers & CounselorsCounselors
Short Guide for teachers Short Guide for teachers (free online)(free online) Profile interpretation system Profile interpretation system (free online)(free online)
NormsNorms
Flynn Effect (next slide)Flynn Effect (next slide) Shaunessy et al. (2004)Shaunessy et al. (2004)
Cattell Culture Fair test 17.8 IQ points Cattell Culture Fair test 17.8 IQ points higher than NNAThigher than NNAT
Project Bright Horizon in PhoenixProject Bright Horizon in Phoenix 2000 K-6 children, about ½ ELL2000 K-6 children, about ½ ELL CogAT, Raven, NNATCogAT, Raven, NNAT Raven 10 SAS points higher than CogAT Raven 10 SAS points higher than CogAT
or NNATor NNAT
70
75
80
85
90
95
100
105
1910 1920 1930 1940 1950 1960 1970 1980 1990 2000
Year
IQ o
n t
he
19
95
Sc
ale
Gains in Wechsler-Binet IQ for the U.S. White population. Sources J. Horgan (1995) and D. Schildlovsky.
Example of Flynn EffectExample of Flynn Effect
Mistakes in norming Mistakes in norming NNATNNAT
NNAT SD's by Test Level
0
5
10
15
20
25
A B C D E F GTest Level
Sta
ndar
d D
evia
tio
n .
George (2001)
Naglieri &Ronning (2000)
Bright Horizon
True Versus Reported NAI Scores True Versus Reported NAI Scores by by NNATNNAT Test Level Test Level
True NAI Score
Level 100 115 130 145
A 100 121 142 163
B 100 119 139 158
C 100 119 137 156
D 100 117 134 151
E 100 115 130 145
F 100 116 132 149
G 100 116 132 148
True Versus Reported NAI Scores True Versus Reported NAI Scores by by NNATNNAT Test Level Test Level
True NAI Score
Level 100 115 130 145
A 100 121 142 163
B 100 119 139 158
C 100 119 137 156
D 100 117 134 151
E 100 115 130 145
F 100 116 132 149
G 100 116 132 148
Over-identification Rates for the Over-identification Rates for the Number Number
of Students with NAI Scores Above 115, of Students with NAI Scores Above 115, 130, and 145130, and 145
True NAI Score
Level 115 130 145
A 1.5 3.4 11.9
B 1.4 2.6 7.3
C 1.3 2.3 5.8
D 1.2 1.7 2.9
E 1.0 1.0 1.0
F 1.1 1.4 2.0
G 1.1 1.4 1.9
Over-identification Rates for the Over-identification Rates for the Number Number
of Students with NAI Scores Above 115, of Students with NAI Scores Above 115, 130, and 145130, and 145
True NAI Score
Level 115 130 145
A 1.5 3.4 11.9
B 1.4 2.6 7.3
C 1.3 2.3 5.8
D 1.2 1.7 2.9
E 1.0 1.0 1.0
F 1.1 1.4 2.0
G 1.1 1.4 1.9
Interpreting Interpreting CogAT scoresCogAT scores
Primary uses of CogATPrimary uses of CogAT
To guide efforts to adapt To guide efforts to adapt instruction to the needs and instruction to the needs and abilities of studentsabilities of students
To provide an alternative measure of To provide an alternative measure of cognitive developmentcognitive development
To identify students whose predicted To identify students whose predicted levels of achievement differ levels of achievement differ markedly from their observed levels markedly from their observed levels of achievementof achievement
Myths about adapting Myths about adapting instructioninstruction
All students are pretty much alikeAll students are pretty much alike
Reading Vocab Across Reading Vocab Across GradesGrades
VOCABULARY
100
150
200
250
300
350
400
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade
Voca
bula
ry D
evel
opm
enta
l Sta
ndar
d S
core
99th %-tile
80th %-tile
50th %-tile
20th %-tile
1st %-tile
Reading Vocab Across GradesReading Vocab Across Grades
VOCABULARY
100
150
200
250
300
350
400
K 1 2 3 4 5 6 7 8 9 10 11 12
Grade
Voca
bula
ry D
evel
opm
enta
l Sta
ndar
d S
core
99th %-tile
80th %-tile
50th %-tile
20th %-tile
1st %-tile
Myths about adapting Myths about adapting instructioninstruction
All students are pretty much alikeAll students are pretty much alike Every student is uniqueEvery student is unique
Myths about adapting Myths about adapting instructioninstruction
All students are pretty much alikeAll students are pretty much alike Every student is uniqueEvery student is unique Adaptations should be based on self-Adaptations should be based on self-
reported learning stylesreported learning styles
Myths about adapting Myths about adapting instructioninstruction
All students are pretty much alikeAll students are pretty much alike Every student is uniqueEvery student is unique Adaptations should be based on self-Adaptations should be based on self-
reported learning stylesreported learning styles If the method is right, the outcome If the method is right, the outcome
will be goodwill be good
Examples of correlationsExamples of correlations
Aspirin and reduced risk of death by heart attacka .02 22,071
General batting skill as a Major League baseball player and hit success on a given instance at bata
.06 —
Calcium intake and bone mass in premenopausal womena
.08 2,493
Effect of nonsteroidal anti-inflammatory drugs (e.g., ibuprofen) on pain reductiona
.14 8,488
Predictor and criterion r N
Weight and height for U.S. adultsa .44 16,948
Myths about adapting Myths about adapting instructioninstruction
All students are pretty much alikeAll students are pretty much alike Every student is uniqueEvery student is unique Adaptations should be based on self-Adaptations should be based on self-
reported learning stylesreported learning styles If the method is right, the outcome If the method is right, the outcome
will be goodwill be good Individualization requires separate Individualization requires separate
learning taskslearning tasks
Important Characteristics Important Characteristics of Studentsof Students
Cognition (knowing)Cognition (knowing) Domain knowledge & skillDomain knowledge & skill Reasoning abilities in the symbol Reasoning abilities in the symbol
systems used to communicate systems used to communicate knowledge knowledge (Verbal, Quant., Spatial)(Verbal, Quant., Spatial)
Affection (feeling)Affection (feeling) anxiety, interests, working alone/with othersanxiety, interests, working alone/with others
Conation (willing)Conation (willing) persistence, impulsivitypersistence, impulsivity
Important Characteristics Important Characteristics of Classroomsof Classrooms
StructureStructure Novelty/Complexity/AbstractnessNovelty/Complexity/Abstractness Dominant symbol systemDominant symbol system Opportunities for working alone or Opportunities for working alone or
with otherswith others
General Principles of Instructional General Principles of Instructional AdaptationAdaptation
Build on StrengthBuild on Strength Focus on working memoryFocus on working memory Scaffold wiselyScaffold wisely Emphasize strategiesEmphasize strategies When grouping, aim for When grouping, aim for
diversitydiversity
Case Study: NaomiCase Study: Naomi
1 25 50 75 99
V 67Q 17N 71
PR
Profile 6E (Q-)
Primary uses of CogATPrimary uses of CogAT
To guide efforts to adapt instruction To guide efforts to adapt instruction to the needs and abilities of studentsto the needs and abilities of students
To provide an alternative To provide an alternative measure of cognitive measure of cognitive developmentdevelopment
To identify students whose predicted To identify students whose predicted levels of achievement differ levels of achievement differ markedly from their observed levels markedly from their observed levels of achievementof achievement
ITBS – CogAT correlationITBS – CogAT correlation
ITBS
Cog
AT
Low High
Low
High
ITBS – CogAT correlationITBS – CogAT correlation
ITBS
Cog
AT
Low High
Low
High
ITBS – CogAT correlationITBS – CogAT correlation
ITBS
Cog
AT
Low High
Low
High
ITBS – CogAT correlationITBS – CogAT correlation
ITBS
Cog
AT
Low High
Low
High
ITBS only
CogAT onlyBoth
Proportion of students Proportion of students identified by one test also identified by one test also
identified by the second testidentified by the second test
Cut scoreCut score
Correlation between testsCorrelation between tests
0.500.50 0.600.60 0.700.70 0.800.800.90.9
00
Top 1%Top 1% 0.130.13 0.190.19 0.270.27 0.380.38 0.540.54
Top 2%Top 2% 0.170.17 0.230.23 0.310.31 0.420.42 0.580.58
Top 3%Top 3% 0.200.20 0.260.26 0.350.35 0.450.45 0.600.60
““Do not use the Composite Do not use the Composite score to screen children for score to screen children for
academic giftedness”academic giftedness”
Thorndike & Hagen (1984) (CogAT4)Thorndike & Hagen (1984) (CogAT4) Thorndike & Hagen (1992) (CogAT5)Thorndike & Hagen (1992) (CogAT5) Lohman & Hagen (2000) (CogAT6)Lohman & Hagen (2000) (CogAT6)
Generally good news for low Generally good news for low achieving studentsachieving students
The lower the student’s score on an The lower the student’s score on an achievement testachievement test
The greater the likelihood that The greater the likelihood that CogAT scores will be higherCogAT scores will be higher
Especially for nonverbal batteryEspecially for nonverbal battery
Primary uses of CogATPrimary uses of CogAT
To guide efforts to adapt instruction To guide efforts to adapt instruction to the needs and abilities of studentsto the needs and abilities of students
To provide an alternative measure of To provide an alternative measure of cognitive developmentcognitive development
To identify students whose predicted To identify students whose predicted levels of achievement differ levels of achievement differ markedly from their observed levels markedly from their observed levels of achievementof achievement
Achievement
PredictedAchievementScore
70 80 90 100 110 120 130 Standard Age Score
Distribution ofAchievementfor SAS of 110
High
Avg
Predicting Achievement from Predicting Achievement from
AbilityAbility
Moderate CorrelationModerate Correlation
Ability
Ach
ieve
men
t
Moderate Correlation
Unexpectedly High Ach.
Unexpectedly Low Ach.
Expected Level of Ach.
A
B
Predicting Ach vrs Flagging Predicting Ach vrs Flagging Ach-Ability discrepanciesAch-Ability discrepancies
Who are the students (at any ach Who are the students (at any ach level) who are most likely to improve level) who are most likely to improve if given new motivation or if given new motivation or instructional resources?instructional resources?
Reasoning Ability > Reasoning Ability > AchievementAchievement
1.1. UnderachievementUnderachievement• poor effort, instruction, etc.poor effort, instruction, etc.
2.2. Well developed ability to transfer Well developed ability to transfer knowledge & skills to novel knowledge & skills to novel situationssituations
• evidence for practice in varied evidence for practice in varied contextscontexts
Achievement > Reasoning Achievement > Reasoning AbilityAbility
1.1. OverachievementOverachievement• unusual effort, good instructionunusual effort, good instruction
2.2. Difficulty in applying Difficulty in applying knowledge/skills in unfamiliar knowledge/skills in unfamiliar contextscontexts
• need for integration, cross-course need for integration, cross-course transfertransfer
General issues in General issues in selectionselection
Golden Rules of selectionGolden Rules of selection
Identification criteria must be Identification criteria must be logically and psychologically tied to logically and psychologically tied to the requirements of the day-to-day the requirements of the day-to-day activities that students will pursue.activities that students will pursue. Mathematics?Mathematics? Literary arts?Literary arts? Visual Arts?Visual Arts?
Differentiated selection implies Differentiated selection implies differentiated instructiondifferentiated instruction
Example r = .6 using PR
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Nonverbal Reasoning
Mat
hem
atic
s A
chie
vem
ent
Example of r = .6Example of r = .6
Example r = .6 using PR
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Nonverbal Reasoning
Mat
hem
atic
s A
chie
vem
ent
Example r = .6 Example r = .6
Example r = .6 using PR
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Nonverbal Reasoning
Mat
hem
atic
s A
chie
vem
ent
Example r = .6Example r = .6
Example r = .6 using PR
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Nonverbal Reasoning
Mat
hem
atic
s A
chie
vem
ent
29%71%
Example r = .6Example r = .6
Imprecision of even high Imprecision of even high correlationscorrelations
Given r = .8Given r = .8 What is the likelihood that a student What is the likelihood that a student
who scores in 60-70who scores in 60-70thth PR at Time 1 PR at Time 1 will scores in the 60-70will scores in the 60-70thth PR at Time PR at Time 2?2?
Lohman, D. F. (2003). Lohman, D. F. (2003). Tables of prediction efficienTables of prediction efficienciescies
. .
Lohman, D. F. (2003). Lohman, D. F. (2003). Tables of prediction efficienTables of prediction efficienciescies
. .
Lohman, D. F. (2003). Lohman, D. F. (2003). Tables of prediction efficienTables of prediction efficienciescies
. .
Proportion of students Proportion of students identified by both testsidentified by both tests
Cut scoreCut score
Correlation between testsCorrelation between tests
0.500.50 0.600.60 0.700.70 0.800.800.90.9
00
Top 1%Top 1% 0.130.13 0.190.19 0.270.27 0.380.38 0.540.54
Top 2%Top 2% 0.170.17 0.230.23 0.310.31 0.420.42 0.580.58
Top 3%Top 3% 0.200.20 0.260.26 0.350.35 0.450.45 0.600.60
Regression to the meanRegression to the mean
The tendency of students with high The tendency of students with high scores to obtain somewhat lower scores to obtain somewhat lower scores upon retestscores upon retest 0 at the mean0 at the mean Increases with distance from the meanIncreases with distance from the mean
Easily predicted from correlationEasily predicted from correlation YYpredpred = Mean + r (Y – mean) = Mean + r (Y – mean)
Causes of Regression to Causes of Regression to the Meanthe Mean
““Errors” of measurementErrors” of measurement Often much larger for high scoring Often much larger for high scoring
studentsstudents Differential growth ratesDifferential growth rates Changes in the abilities measured by Changes in the abilities measured by
the tests at time 1 and time 2 (esp the tests at time 1 and time 2 (esp achievement tests)achievement tests)
Changes in the norming populationChanges in the norming population school sample or national age sampleschool sample or national age sample
Reducing RegressionReducing Regression
Use the most reliable tests available Use the most reliable tests available (judge by SEM on reported score scale)(judge by SEM on reported score scale)
Avoid accepting the highest score as Avoid accepting the highest score as the best estimate of abilitythe best estimate of ability
Average scoresAverage scores Ability and Achievement test scoresAbility and Achievement test scores
Within domain (e.g., math ach & CogAT Q or Within domain (e.g., math ach & CogAT Q or QN)QN)
Achievement at T1 and T2Achievement at T1 and T2 Revolving door policiesRevolving door policies
Combining Combining scoresscores
"And" "Or" "Average"
Test 1 and Test 2 Test 1 or Test 2 Average of Test 1 and Test 2
Figure 5. Plots of the effects of three rules: (a) high scores on test 1 and test 2; (b) high
scores on test 1 or test 2; and (c) high scores on the average of test 1 and test 2.
““And,” “or” or “Average”And,” “or” or “Average”
Screening testsScreening tests
You administer a screening test to You administer a screening test to reduce the number who must be reduce the number who must be administered the admissions testadministered the admissions test
Assume a correlation of r = .6 between Assume a correlation of r = .6 between the two teststhe two tests
Assume students must score at the 95Assume students must score at the 95thth PR or higher on the admissions testPR or higher on the admissions test
What cut score on the screening test What cut score on the screening test will include all of those who would will include all of those who would meet this criterion?meet this criterion?
Proportion of students in top X percent of Proportion of students in top X percent of screening test who exceed the same or a more screening test who exceed the same or a more
stringent cut score on follow up teststringent cut score on follow up test r = .6
Admissions test
Top x % 5% 3% 1%
Scree
nin
g T
est
30% 0.80 0.84 0.91
25% 0.75 0.80 0.87
20% 0.68 0.73 0.82
15% 0.59 0.65 0.75
10% 0.48 0.54 0.65
5% 0.31 0.36 0.48
3% 0.22 0.26 0.36
Proportion of students in top X percent of Proportion of students in top X percent of screening test who exceed the same or a more screening test who exceed the same or a more
stringent cut score on follow up teststringent cut score on follow up test r = .6
Admissions test
Top x % 5% 3% 1%
Scree
nin
g T
est
30% 0.80 0.84 0.91
25% 0.75 0.80 0.87
20% 0.68 0.73 0.82
15% 0.59 0.65 0.75
10% 0.48 0.54 0.65
5% 0.31 0.36 0.48
3% 0.22 0.26 0.36
Screening might make Screening might make sensesense
When admissions test is expensive to When admissions test is expensive to administeradminister
When the correlation between the When the correlation between the admissions & screening test is very admissions & screening test is very highhigh
When there are many more applicants When there are many more applicants than places in the programthan places in the program
When the false rejection rate is not an When the false rejection rate is not an issue issue
Local versus National Local versus National NormsNorms
Except for regional or national talent Except for regional or national talent searches, the PRIMARY reference group is searches, the PRIMARY reference group is not the nation or even the state but the not the nation or even the state but the school or school district.school or school district.
The need for special instruction depends on The need for special instruction depends on the discrepancy between the child’s level of the discrepancy between the child’s level of cognitive and academic development and cognitive and academic development and that of his or her classmates.that of his or her classmates.
Multiple perspectives: Nation, the local Multiple perspectives: Nation, the local population, opportunity-to-learn subgroups population, opportunity-to-learn subgroups within the local populationwithin the local population
Identification of Identification of Talent in Special Talent in Special
PopulationsPopulations
ELL childrenELL children
Identifying academic Identifying academic talenttalent
Not giftednessNot giftedness
Tradeoff Tradeoff
Measuring the right things Measuring the right things approximately for ELL students approximately for ELL students
or or
the wrong things with greater the wrong things with greater accuracyaccuracy
Inference of Aptitude?Inference of Aptitude?
When someone learns in a few When someone learns in a few trials what others learn in many trials what others learn in many trialstrials
Opportunity to learn is criticalOpportunity to learn is critical Common norms appropriate only Common norms appropriate only
if experiences are similarif experiences are similar Placement by achievementPlacement by achievement
Multiple PerspectivesMultiple Perspectives The need for special programming depends The need for special programming depends
most importantly on the discrepancy most importantly on the discrepancy between a child’s achievements & abilities between a child’s achievements & abilities and that of his or her and that of his or her classmatesclassmates
Except for regional talent searches, summer Except for regional talent searches, summer programs that draw from different schools, programs that draw from different schools, etc… Make better use of local norms!etc… Make better use of local norms!
For ELL students in grade 3, compare scores For ELL students in grade 3, compare scores to:to: Other ELL students in grade 3Other ELL students in grade 3 Other students in grade 3 in the district/schoolOther students in grade 3 in the district/school Other grade 3 students in the nationOther grade 3 students in the nation
Multiple Programming Multiple Programming OptionsOptions
Current level of achievement is primary Current level of achievement is primary guideguide
Programming goal: to improve the Programming goal: to improve the achievement at a rate faster than would achievement at a rate faster than would otherwise occurotherwise occur
For on- and below-grade-level For on- and below-grade-level achievement options include: tutors, after-achievement options include: tutors, after-school or weekend classes/clubs, etc. school or weekend classes/clubs, etc. Motivational component critical.Motivational component critical.
For achievement well in advance of peers, For achievement well in advance of peers, consider single-subject accelerationconsider single-subject acceleration
Combining ITBS and Combining ITBS and CogATCogAT
Grades K – 2Grades K – 2 Average CogAT V and ITBS Reading TotalAverage CogAT V and ITBS Reading Total Average CogAT Q and ITBS Math totalAverage CogAT Q and ITBS Math total CogAT NV stands aloneCogAT NV stands alone
Grades 3 – 12Grades 3 – 12 Average CogAT V and ITBS Reading TotalAverage CogAT V and ITBS Reading Total Average CogAT QN and ITBS Math TotalAverage CogAT QN and ITBS Math Total
Use NCE scores – they can be averagedUse NCE scores – they can be averaged Then sort by grade and OTL groupThen sort by grade and OTL group
Integrating ability, Integrating ability, achievement, and teacher achievement, and teacher
ratingsratings See Lohman, D. F. & Renzulli, J. See Lohman, D. F. & Renzulli, J.
(2007). (2007). A simple procedure for combining abA simple procedure for combining ability test scores, achievement test scility test scores, achievement test scores, and teacher ratings to identify ores, and teacher ratings to identify academically talented children. academically talented children.
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
Creativity (NOMINATED students only)Creativity (NOMINATED students only)
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
(ALL (ALL Students)Students)
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Below Avg.Below Avg. Above Avg.Above Avg.
>97>97thth PR PR
>80>80thth PR PR
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Below Avg.Below Avg. Above Avg.Above Avg.
>97>97thth PR PR IIII II
>80>80thth PR PR IVIV IIIIII
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Below Avg.Below Avg. Above Avg.Above Avg.
>97>97thth PR PR IIII IIadmitadmit
>80>80thth PR PR IVIV IIIIII
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Below Avg.Below Avg. Above Avg.Above Avg.
>97>97thth PR PR IIIIAdmit but Admit but
watchwatch
II
>80>80thth PR PR IVIV IIIIII
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Below Avg.Below Avg. Above Avg.Above Avg.
>97>97thth PR PR IIII II
>80>80thth PR PR IVIV IIIIIIEnrichmentEnrichment
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Below Avg.Below Avg. Above Avg.Above Avg.
>97>97thth PR PR IIII II
>80>80thth PR PR IVIVTry next yearTry next year
IIIIII
Verbal Verbal AbilityAbility
Or Or
Quant/NVQuant/NVAbilityAbility
Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or
CreativityCreativity
Below Avg.Below Avg. Above Avg.Above Avg.
>97>97thth PR PR IIIIAdmit but Admit but
watchwatch
IIadmitadmit
>80>80thth PR PR IVIVTry next yearTry next year
IIIIIIEnrichmentEnrichment
Final Thoughts: Using Final Thoughts: Using CogATCogAT
Examine warnings and confidence intervals Examine warnings and confidence intervals on score reportson score reports
Do not screen using Composite scoreDo not screen using Composite score Use V and QN instead (at grade 3 +)Use V and QN instead (at grade 3 +) Combine with Reading Total and Math TotalCombine with Reading Total and Math Total
AverageAverage measures of the same construct; measures of the same construct; Use “or” for measures of different constructsUse “or” for measures of different constructs
To identify talent, measure the right To identify talent, measure the right aptitudes but then compare scores to the aptitudes but then compare scores to the proper norm group(s)proper norm group(s)
Emphasize local norms for in-school Emphasize local norms for in-school programsprograms
ELLELL Compare the performance of the ELL 3Compare the performance of the ELL 3rdrd
grader with that of other ELL 3grader with that of other ELL 3rdrd graders graders Be wary of national norms that you can Be wary of national norms that you can
purchase– esp on nonverbal tests (Raven, purchase– esp on nonverbal tests (Raven, NNAT,…) NNAT,…)
Nonverbal tests have a role to play, but Nonverbal tests have a role to play, but should never stand aloneshould never stand alone
Emphasize the identification of talent Emphasize the identification of talent rather than the identification of rather than the identification of giftednessgiftedness
It is unwise to accept the highest It is unwise to accept the highest score as the best estimate of abilityscore as the best estimate of ability
Combine ability and achievement test Combine ability and achievement test scores in principled ways scores in principled ways
Teacher ratings are only as good as Teacher ratings are only as good as teacher training in making ratingsteacher training in making ratings
Do not simply add teacher ratings and Do not simply add teacher ratings and similar measures to similar measures to ability/achievement scoresability/achievement scores
GeneralGeneral
There is no way to measure innate There is no way to measure innate ability; all abilities are developedability; all abilities are developed
Measures of achievement and ability Measures of achievement and ability differ in degree – not kinddiffer in degree – not kind
Future expertise is built on the base of Future expertise is built on the base of current knowledge in a domain, current knowledge in a domain, reasoning abilities needed for new reasoning abilities needed for new learning in that domain, interest in the learning in that domain, interest in the domain, and the ability to persist in the domain, and the ability to persist in the pursuit of excellence pursuit of excellence
All of which depend on opportunity and All of which depend on opportunity and circumstancecircumstance
The EndThe End
www.cogat.comwww.cogat.com
http://faculty.education.uiowa.edu/http://faculty.education.uiowa.edu/dlohmandlohman
NCE ScoresNCE Scores
Get from the publisher for CogATGet from the publisher for CogAT Table look up (Table 32 in CogAT Table look up (Table 32 in CogAT
Norms Manual)Norms Manual) Convert PR’s to NCE scoresConvert PR’s to NCE scores
In Excel In Excel NCE = NORMINV (PR/100, 50, 21.06)NCE = NORMINV (PR/100, 50, 21.06)
If SAS > 135If SAS > 135 NCE = 21.06 * [(SAS – 100)/16] + 50NCE = 21.06 * [(SAS – 100)/16] + 50