Using CogAT David Lohman Institute for Research and Policy on Acceleration Belin-Blank Center & Iowa Testing Programs University of Iowa

Using CogATUsing CogAT

David LohmanDavid LohmanInstitute for Research and Policy on Institute for Research and Policy on

AccelerationAccelerationBelin-Blank CenterBelin-Blank Center

&&Iowa Testing ProgramsIowa Testing Programs

University of IowaUniversity of Iowa

http://faculty.education.uiowa.edu/dlohman/

TopicsTopics Distinguishing between ability & Distinguishing between ability &

achievementachievement Overview of CogATOverview of CogAT Comparing CogAT with other ability testsComparing CogAT with other ability tests Interpreting CogAT scoresInterpreting CogAT scores General issues in selectionGeneral issues in selection Identification of talent in special Identification of talent in special

populationspopulations Combining Achievement, Ability, & Teacher Combining Achievement, Ability, & Teacher

ratings: the Lohman – Renzulli matrixratings: the Lohman – Renzulli matrix

Distinguishing Distinguishing between ability and between ability and

achievementachievement

Puzzlements for common Puzzlements for common interpretations of ability & interpretations of ability &

achievementachievement Is ability more biologically based?Is ability more biologically based?

Most studies show same heritability for IQ (Gf) Most studies show same heritability for IQ (Gf) and achievement tests (Gc)and achievement tests (Gc)

Lower relative achievement than ability = Lower relative achievement than ability = underachievementunderachievement But there are an equal number of “overachievers”But there are an equal number of “overachievers”

Status scores (IQ, PR) show good stabilityStatus scores (IQ, PR) show good stability But one must keep getting better to retain that IQBut one must keep getting better to retain that IQ Between 9 – 17 r(True IQ) = .75. Between 9 – 17 r(True IQ) = .75.

60% in top 3% at 9 NOT in top 3% at age 1760% in top 3% at 9 NOT in top 3% at age 17

Fluid abilities invested in experience Fluid abilities invested in experience to produce particular constellations to produce particular constellations of crystallized abilities?of crystallized abilities? Only for very young childrenOnly for very young children Thereafter, crystallized abilities -> fluidThereafter, crystallized abilities -> fluid

Level 1. NominalismLevel 1. Nominalism (Most (Most people here)people here)

““ability” and “achievement” are ability” and “achievement” are separate (Jangle fallacy –T. separate (Jangle fallacy –T.

Kelley, 1927)Kelley, 1927)

AbilityAbility AchievementAchievement

Ability Achievement

Level 2. Oh, Oh – there’s more Level 2. Oh, Oh – there’s more overlap than uniqueness here!overlap than uniqueness here!

Its all ‘g’ (any indicant will do)Its all ‘g’ (any indicant will do) Its all just a product of experienceIts all just a product of experience Preserve stage 1 beliefs –Preserve stage 1 beliefs –

Purge ability of visible achievement (e.g. Purge ability of visible achievement (e.g. measure “process” or use only “nonverbal” measure “process” or use only “nonverbal” measures)measures)

Level 3. Island kingdoms –Things Level 3. Island kingdoms –Things

get even more complicatedget even more complicated ((most most scholars of human abilities)scholars of human abilities) Effects of language, culture, and Effects of language, culture, and

experience on the development of experience on the development of ability (“All abilities are developed” ability (“All abilities are developed” Anastasi)Anastasi)

Experience alters the structure of Experience alters the structure of the brainthe brain

Mental processes do not exist Mental processes do not exist independently of knowledge. independently of knowledge.

70

75

80

85

90

95

100

105

1910 1920 1930 1940 1950 1960 1970 1980 1990 2000

Year

IQ o

n t

he

19

95

Sc

ale

Gains in Wechsler-Binet IQ for the U.S. White population. Sources J. Horgan (1995) and D. Schildlovsky.

Example of Flynn EffectExample of Flynn Effect

Proportion of variance in WISC Full Scale IQ at age 7 accounted for by genetic factors as a function of socioeconomic status (SES)

Turkheimer et al. (2003) Psychological Science, 14 (6). N= 319 twin pairs.43% White, 54% Black. Most families poor.

Low High

Fluid-Crystallized Continuum (1)Fluid-Crystallized Continuum (1)

Fluid Crystallized

Physical skills

General physical fitness

Basketball

Swimming

Football

Field hockey

Volleyball

Wrestling Cycling


Fluid Crystallized

Cognitive abilities

Physical skills

General fluid ability (Gf)

Science achievement

Math achievement

Social studies achievement

Knowledge of literature

Specific factual

knowledge


Basketball

Swimming

Football

Field hockey

Volleyball

Wrestling Cycling


Fluid Crystallized

Cognitive abilities

Physical skills

General fluid ability (Gf)

Science achievement

Math achievement

Social studies achievement

Knowledge of literature

Specific factual

knowledge


Basketball

Swimming

Football

Field hockey

Volleyball

Wrestling Cycling

A common ability-achievement A common ability-achievement spacespace

Level 4. Systems theoriesLevel 4. Systems theories (A (A handful)handful)

Aptitude Theory (Richard Snow)Aptitude Theory (Richard Snow)

Sidesteps the issue of defining Sidesteps the issue of defining intelligence;intelligence;

starts with expertise & the starts with expertise & the contexts in which it is developed & contexts in which it is developed & displayed, displayed,

readiness to learn in those readiness to learn in those contextscontexts

Overview of Overview of CogATCogAT

Some HistorySome History

Lorge -Thorndike Intelligence testLorge -Thorndike Intelligence test Cognitive AbilitieCognitive Abilitiess Test Test

Form 1 1974Form 1 1974 Forms 2 – 3 (no Composite score)Forms 2 – 3 (no Composite score) Forms 4 – Thorndike & Hagen – Comp Forms 4 – Thorndike & Hagen – Comp

scorescore Form 5 – HagenForm 5 – Hagen Form 6 – Lohman & HagenForm 6 – Lohman & Hagen

Co-normed with the ITBS & ITEDCo-normed with the ITBS & ITED

Primary uses of CogATPrimary uses of CogAT

To guide efforts to adapt instruction To guide efforts to adapt instruction to the needs and abilities of studentsto the needs and abilities of students

To provide an alternative measure of To provide an alternative measure of cognitive developmentcognitive development

To identify students whose predicted To identify students whose predicted levels of achievement differ levels of achievement differ markedly from their observed levels markedly from their observed levels of achievementof achievement

Primary Battery (K-2)Primary Battery (K-2)

Oral VocabularyVerbal Reasoning

Verbal Reasoning.....

Relational ConceptsQuantitative Concepts

Quantitative Reasoning.....

Figure ClassificationMatrices

Nonverbal Reasoning.....

General Reasoning Ability

No readingTests untimed (paced by teacher)Mark directly in booklet

Multilevel Battery (gr. 3-Multilevel Battery (gr. 3-12)12)

Verbal ClassificationSentence Completion

Verbal Analogies

Verbal Reasoning......

Quantitative RelationsNumber Series

Equation Building

Quantitative Reasoning.....

Figure ClassificationFigure AnalogiesFigure Analysis

Nonverbal Reasoning.....

General Reasoning Ability

Tests timedSeparate Answer sheetCommon Directions

3 Separate Test 3 Separate Test BatteriesBatteries

(Not one)(Not one)

ScoresScores

Raw score = number correctRaw score = number correct Scale score – USSScale score – USS

Within level - map number correct on to Within level - map number correct on to a scale whose intervals are a scale whose intervals are approximately the same sizeapproximately the same size

Between levels – maps number correct Between levels – maps number correct on different levels of the test on to a on different levels of the test on to a single, common, developmental scalesingle, common, developmental scale

A

USS Scale

etc

B

C

D

Relationships among Stanines, Percentile Relationships among Stanines, Percentile Ranks, and Standard Age ScoresRanks, and Standard Age Scores

134 - 150

CompositesComposites

Composite scoresComposite scores Partial VQ, VN, QNPartial VQ, VN, QN Full – VQN or C [do NOT use for Full – VQN or C [do NOT use for

screening]screening] Primary BatteryPrimary Battery

V or (VQ) versus NV or (VQ) versus N Multilevel BatteryMultilevel Battery

V versus QNV versus QN

Consequential Validity:Consequential Validity:Score warnings Score warnings

Age out of rangeAge out of range Age unusual for coded gradeAge unusual for coded grade Estimated test levelEstimated test level Level unusual for coded gradeLevel unusual for coded grade Targeted scoreTargeted score Too few items attempted to scoreToo few items attempted to score Many items omitted (slow and Many items omitted (slow and

accurate)accurate) Extremely variable responsesExtremely variable responses

Personal Confidence Personal Confidence IntervalsIntervals

Pattern of item responses aberrant?Pattern of item responses aberrant? Inconsistent across subtests within a battery?Inconsistent across subtests within a battery? Personal Standard Error of Measurement Personal Standard Error of Measurement

(PSEM)(PSEM)

1 25 50 75 99

V 120 89Q 116 84N 125 94

SAS PR

Score ProfilesScore Profiles

CogAT 6 ‘ABC’ Profile CogAT 6 ‘ABC’ Profile systemsystem

Measuring the patternMeasuring the pattern ““A” profiles:A” profiles: Confidence bands overlap for Confidence bands overlap for

all three scores. Scores are at roughly the all three scores. Scores are at roughly the ssAAmeme level level

““B” profiles:B” profiles: One score is One score is aaBBoveove or or BBelowelow the other two scores, which do not differthe other two scores, which do not differ

““C” profiles:C” profiles: Two scores Two scores CContrastontrast

““E” profiles:E” profiles: Extreme B or C profiles Extreme B or C profiles (>=24)(>=24)

““A” ProfileA” Profile

1 25 50 75 99

V 120 89Q 116 84N 125 94

SAS PR

““B” ProfilesB” Profiles

1 25 50 75 99

V 120 89Q 116 84N 100 50

SAS PR

1 25 50 75 99

V 95 38Q 92 31N 110 73

SAS PR

N-

N+

““C” ProfileC” Profile

1 25 50 75 99

V 120 89Q 110 73N 100 50

SAS PR

V+ N-

Extreme “C” ProfileExtreme “C” Profile

1 25 50 75 99

V 120 89Q 107 67N 92 31

SAS PR

SAS Max – SAS Min = 28 E (V+ N-)

Profile LevelProfile Level

Median (middle) age stanineMedian (middle) age stanine

66 A A

55 B (V+) B (V+)

88 C (Q+ V-) C (Q+ V-)

22 E (N+ V-)E (N+ V-)

CogAT6 Profile frequencies for CogAT6 Profile frequencies for students students

in K-12 populationin K-12 population

ProfileProfile

Percent Percent in K-12 in K-12 populatipopulati

onon

AA 3333

BB 4242

B+B+ ( 21)( 21)

B -B - (22)(22)

EE 77

B+B+ (4)(4)

B -B - (3)(3)

CogAT6 Profile frequencies for CogAT6 Profile frequencies for students in K-12 pop. and for students students in K-12 pop. and for students

with two stanine scores of 9with two stanine scores of 9

ProfileProfile

Percent Percent in K-12 in K-12 populatipopulati

onon

Percent Percent in in

Stanine=Stanine=9 group9 group

AA 3333 3737

BB 4242 2727

B+B+ ( 21)( 21) ( 6)( 6)

B -B - (22)(22)

(( 21)21)

EE 77 1919

B+B+ 44 ( 3)( 3)

B -B - 33 ( ( 16)16)

37%

Comparing Comparing CogAT with other CogAT with other

teststests

ReliabilityReliability Many estimates for a given testMany estimates for a given test

Sources of errorSources of error Correlation versus standard error of Correlation versus standard error of

measurement (SEM)measurement (SEM) Correlations depend on sample variabilityCorrelations depend on sample variability Easily misinterpretedEasily misinterpreted

SEMSEM Typical SD of distribution of test scores if Typical SD of distribution of test scores if

each student could be tested many timeseach student could be tested many times

Person-level estimate – Only on CogATPerson-level estimate – Only on CogAT

SEM for SAS scoresSEM for SAS scores




Standard Errors of Standard Errors of Measurement for Individual Measurement for Individual

& Group Tests& Group Tests

WISCWISC-IV-IV SB-VSB-V

CogAT CogAT 66

OLSAT-OLSAT-88 InviewInview RavenRaven NNATNNAT

VerbalVerbal 3.93.9 3.63.6 3.43.4 5.75.7 5.35.3

NonverbalNonverbal 4.24.2 3.93.9 3.73.7 5.85.8 4.54.5 3.03.0 6.16.1

QuantitativeQuantitative 4.54.5 5.35.3 3.33.3

Comp/Full ScaleComp/Full Scale 2.82.8 2.82.8 2.22.2 5.75.7 3.53.5

Standard Errors of Standard Errors of Measurement for Individual Measurement for Individual

& Group Tests& Group Tests

WISCWISC-IV-IV SB-VSB-V

CogAT CogAT 66

OLSAT-OLSAT-88 InviewInview RavenRaven NNATNNAT

VerbalVerbal 3.93.9 3.63.6 3.43.4 5.75.7 5.35.3

NonverbalNonverbal 4.24.2 3.93.9 3.73.7 5.85.8 4.54.5 3.03.0 6.16.1

QuantitativeQuantitative 4.54.5 5.35.3 3.33.3

Comp/Full ScaleComp/Full Scale 2.82.8 2.82.8 2.22.2 5.75.7 3.53.5

Individually-administered tests:Individually-administered tests: SB-V SB-V WISC-IVWISC-IV

Group-administered tests:Group-administered tests: Inview Inview Otis-LenonOtis-Lenon NNATNNAT

CogAT is more reliable CogAT is more reliable thanthan

Conditional Standard Error Conditional Standard Error of Meas.of Meas.

Cogat 6 Verbal Battery: Level A

0.00

5.00

10.00

15.00

20.00

0 5 10 15 20 25 30 35 40 45 50 55 60 65

Number Correct

SE

M USS Score

Raw Score

Conditional SEM's for CogAT6 Verbal USS scores, by test level

Verbal USS K 1 2 A B C D . . .. . . . . . .. . . . . . .

191-195 11.5 9.9 6.5 5.3 4.8 4.3196-200 15.9 11.4 7.5 5.9 5.2 4.5201-205 12.0 12.5 8.5 7.0 5.4 4.8206-210 10.5 7.4 5.9 5.2211-215 17.0 12.5 13.0 13.1 8.9 6.9 5.6216-220 13.4 10.4 8.4 6.2221-225 13.0 14.8 13.2 10.9 7.4226-230 13.9 8.5231-235 14.5 14.8 13.3 10.8236-240 15.0 13.3241-245 16.9 14.3246-250251-255 16.5 14.8256-260 95th PR261-265 15.4266-270 99th PR 16.4

Out of level testing?Out of level testing?

SAS or PR scores?SAS or PR scores? Primary Battery – Multilevel Primary Battery – Multilevel

Battery?Battery? Requires individual testingRequires individual testing Assumes child can use machine-Assumes child can use machine-

readable answer sheetreadable answer sheet Quant battery assumes familiarity with Quant battery assumes familiarity with

numerical operationsnumerical operations Level A – H?Level A – H?

Common time limits & directionsCommon time limits & directions

ValidityValidity Construct Construct

Representation --- all three aspects of Representation --- all three aspects of fluid reasoning abilityfluid reasoning ability

Predictive Predictive Excellent for predicting current and Excellent for predicting current and

future academic achievementfuture academic achievement Predictions the same for all ethnic Predictions the same for all ethnic

groups groups ConsequentialConsequential

No other test comes closeNo other test comes close

Validity:Validity:Construct Construct

RepresentationRepresentation

Carroll’s Three-Stratum Theory of Human Abilities

Gf Fluid Reasoning Abilities

Carroll’s Three-Stratum Theory of Human Abilities

VerbalSequentialReasoning

QuantitativeReasoning

Figural-Inferential Reasoning

Correlation between WISC Full Scale Score and CogAT Composite = .79

Predictive ValidityPredictive Validity Correlations with current and subsequent Correlations with current and subsequent

achievementachievement Within Battery predictions strongWithin Battery predictions strong

Verbal with Reading, Soc Studies (r =.4 - .8)Verbal with Reading, Soc Studies (r =.4 - .8) Quant with Mathematics (r = .4 - .8)Quant with Mathematics (r = .4 - .8) Figural–Nonverbal with Math (r = .4 - .7)Figural–Nonverbal with Math (r = .4 - .7)

Negative for verbal ach. after controlling gNegative for verbal ach. after controlling g Across batteriesAcross batteries

Multiple correlations – typically R = .8Multiple correlations – typically R = .8 Often better than prior achievement in the domainOften better than prior achievement in the domain V and QN partial composite especially usefulV and QN partial composite especially useful

Within ethnic-group correlations the sameWithin ethnic-group correlations the same Implications for TALENT identificationImplications for TALENT identification

Consequential Validity: Consequential Validity: Advice on score Advice on score interpretation?interpretation?

Early 20Early 20thth century theory of ‘culture-fair century theory of ‘culture-fair measure of g’ measure of g’

2121stst century theory of reasoning abilities century theory of reasoning abilities Evidence from research on human abilitiesEvidence from research on human abilities Evidence from predictions of academic Evidence from predictions of academic

achievementachievement Evidence from ATI researchEvidence from ATI research Evidence from cognitive psychologyEvidence from cognitive psychology

Consequential Validity: Consequential Validity: Score useScore use

Does Does everyevery child (teacher) receive child (teacher) receive potentially useful information?potentially useful information?

Specific suggestions for how to use the level Specific suggestions for how to use the level and profile of scores to and profile of scores to Assist the child in learning by adapting Assist the child in learning by adapting

instruction better to meet his/her learning styleinstruction better to meet his/her learning style Build on cognitive strengthsBuild on cognitive strengths Shore up weaknessesShore up weaknesses

Interpretive Guide for Teachers & Interpretive Guide for Teachers & CounselorsCounselors

Short Guide for teachers Short Guide for teachers (free online)(free online) Profile interpretation system Profile interpretation system (free online)(free online)

NormsNorms

Flynn Effect (next slide)Flynn Effect (next slide) Shaunessy et al. (2004)Shaunessy et al. (2004)

Cattell Culture Fair test 17.8 IQ points Cattell Culture Fair test 17.8 IQ points higher than NNAThigher than NNAT

Project Bright Horizon in PhoenixProject Bright Horizon in Phoenix 2000 K-6 children, about ½ ELL2000 K-6 children, about ½ ELL CogAT, Raven, NNATCogAT, Raven, NNAT Raven 10 SAS points higher than CogAT Raven 10 SAS points higher than CogAT

or NNATor NNAT

70

75

80

85

90

95

100

105

1910 1920 1930 1940 1950 1960 1970 1980 1990 2000

Year

IQ o

n t

he

19

95

Sc

ale

Gains in Wechsler-Binet IQ for the U.S. White population. Sources J. Horgan (1995) and D. Schildlovsky.

Example of Flynn EffectExample of Flynn Effect

Mistakes in norming Mistakes in norming NNATNNAT

NNAT SD's by Test Level

0

5

10

15

20

25

A B C D E F GTest Level

Sta

ndar

d D

evia

tio

n .

George (2001)

Naglieri &Ronning (2000)

Bright Horizon

True Versus Reported NAI Scores True Versus Reported NAI Scores by by NNATNNAT Test Level Test Level

True NAI Score

Level 100 115 130 145

A 100 121 142 163

B 100 119 139 158

C 100 119 137 156

D 100 117 134 151

E 100 115 130 145

F 100 116 132 149

G 100 116 132 148

True Versus Reported NAI Scores True Versus Reported NAI Scores by by NNATNNAT Test Level Test Level

True NAI Score

Level 100 115 130 145

A 100 121 142 163

B 100 119 139 158

C 100 119 137 156

D 100 117 134 151

E 100 115 130 145

F 100 116 132 149

G 100 116 132 148

Over-identification Rates for the Over-identification Rates for the Number Number

of Students with NAI Scores Above 115, of Students with NAI Scores Above 115, 130, and 145130, and 145

True NAI Score

Level 115 130 145

A 1.5 3.4 11.9

B 1.4 2.6 7.3

C 1.3 2.3 5.8

D 1.2 1.7 2.9

E 1.0 1.0 1.0

F 1.1 1.4 2.0

G 1.1 1.4 1.9

Over-identification Rates for the Over-identification Rates for the Number Number

of Students with NAI Scores Above 115, of Students with NAI Scores Above 115, 130, and 145130, and 145

True NAI Score

Level 115 130 145

A 1.5 3.4 11.9

B 1.4 2.6 7.3

C 1.3 2.3 5.8

D 1.2 1.7 2.9

E 1.0 1.0 1.0

F 1.1 1.4 2.0

G 1.1 1.4 1.9

Interpreting Interpreting CogAT scoresCogAT scores


To guide efforts to adapt To guide efforts to adapt instruction to the needs and instruction to the needs and abilities of studentsabilities of students



Myths about adapting Myths about adapting instructioninstruction

All students are pretty much alikeAll students are pretty much alike

Reading Vocab Across Reading Vocab Across GradesGrades

VOCABULARY

100

150

200

250

300

350

400

K 1 2 3 4 5 6 7 8 9 10 11 12

Grade

Voca

bula

ry D

evel

opm

enta

l Sta

ndar

d S

core

99th %-tile

80th %-tile

50th %-tile

20th %-tile

1st %-tile

Reading Vocab Across GradesReading Vocab Across Grades

VOCABULARY

100

150

200

250

300

350

400

K 1 2 3 4 5 6 7 8 9 10 11 12

Grade

Voca

bula

ry D

evel

opm

enta

l Sta

ndar

d S

core

99th %-tile

80th %-tile

50th %-tile

20th %-tile

1st %-tile


All students are pretty much alikeAll students are pretty much alike Every student is uniqueEvery student is unique


All students are pretty much alikeAll students are pretty much alike Every student is uniqueEvery student is unique Adaptations should be based on self-Adaptations should be based on self-

reported learning stylesreported learning styles



reported learning stylesreported learning styles If the method is right, the outcome If the method is right, the outcome

will be goodwill be good

Examples of correlationsExamples of correlations

Aspirin and reduced risk of death by heart attacka .02 22,071

General batting skill as a Major League baseball player and hit success on a given instance at bata

.06 —

Calcium intake and bone mass in premenopausal womena

.08 2,493

Effect of nonsteroidal anti-inflammatory drugs (e.g., ibuprofen) on pain reductiona

.14 8,488

Predictor and criterion r N

Weight and height for U.S. adultsa .44 16,948



reported learning stylesreported learning styles If the method is right, the outcome If the method is right, the outcome

will be goodwill be good Individualization requires separate Individualization requires separate

learning taskslearning tasks

Important Characteristics Important Characteristics of Studentsof Students

Cognition (knowing)Cognition (knowing) Domain knowledge & skillDomain knowledge & skill Reasoning abilities in the symbol Reasoning abilities in the symbol

systems used to communicate systems used to communicate knowledge knowledge (Verbal, Quant., Spatial)(Verbal, Quant., Spatial)

Affection (feeling)Affection (feeling) anxiety, interests, working alone/with othersanxiety, interests, working alone/with others

Conation (willing)Conation (willing) persistence, impulsivitypersistence, impulsivity

Important Characteristics Important Characteristics of Classroomsof Classrooms

StructureStructure Novelty/Complexity/AbstractnessNovelty/Complexity/Abstractness Dominant symbol systemDominant symbol system Opportunities for working alone or Opportunities for working alone or

with otherswith others

General Principles of Instructional General Principles of Instructional AdaptationAdaptation

Build on StrengthBuild on Strength Focus on working memoryFocus on working memory Scaffold wiselyScaffold wisely Emphasize strategiesEmphasize strategies When grouping, aim for When grouping, aim for

diversitydiversity

Case Study: NaomiCase Study: Naomi

1 25 50 75 99

V 67Q 17N 71

PR

Profile 6E (Q-)



To provide an alternative To provide an alternative measure of cognitive measure of cognitive developmentdevelopment


ITBS – CogAT correlationITBS – CogAT correlation

ITBS

Cog

AT

Low High

Low

High


ITBS

Cog

AT

Low High

Low

High


ITBS

Cog

AT

Low High

Low

High


ITBS

Cog

AT

Low High

Low

High

ITBS only

CogAT onlyBoth

Proportion of students Proportion of students identified by one test also identified by one test also

identified by the second testidentified by the second test

Cut scoreCut score

Correlation between testsCorrelation between tests

0.500.50 0.600.60 0.700.70 0.800.800.90.9

00

Top 1%Top 1% 0.130.13 0.190.19 0.270.27 0.380.38 0.540.54

Top 2%Top 2% 0.170.17 0.230.23 0.310.31 0.420.42 0.580.58

Top 3%Top 3% 0.200.20 0.260.26 0.350.35 0.450.45 0.600.60

““Do not use the Composite Do not use the Composite score to screen children for score to screen children for

academic giftedness”academic giftedness”

Thorndike & Hagen (1984) (CogAT4)Thorndike & Hagen (1984) (CogAT4) Thorndike & Hagen (1992) (CogAT5)Thorndike & Hagen (1992) (CogAT5) Lohman & Hagen (2000) (CogAT6)Lohman & Hagen (2000) (CogAT6)

Generally good news for low Generally good news for low achieving studentsachieving students

The lower the student’s score on an The lower the student’s score on an achievement testachievement test

The greater the likelihood that The greater the likelihood that CogAT scores will be higherCogAT scores will be higher

Especially for nonverbal batteryEspecially for nonverbal battery





Achievement

PredictedAchievementScore

70 80 90 100 110 120 130 Standard Age Score

Distribution ofAchievementfor SAS of 110

High

Avg

Predicting Achievement from Predicting Achievement from

AbilityAbility

Moderate CorrelationModerate Correlation

Ability

Ach

ieve

men

t

Moderate Correlation

Unexpectedly High Ach.

Unexpectedly Low Ach.

Expected Level of Ach.

A

B

Predicting Ach vrs Flagging Predicting Ach vrs Flagging Ach-Ability discrepanciesAch-Ability discrepancies

Who are the students (at any ach Who are the students (at any ach level) who are most likely to improve level) who are most likely to improve if given new motivation or if given new motivation or instructional resources?instructional resources?

Reasoning Ability > Reasoning Ability > AchievementAchievement

1.1. UnderachievementUnderachievement• poor effort, instruction, etc.poor effort, instruction, etc.

2.2. Well developed ability to transfer Well developed ability to transfer knowledge & skills to novel knowledge & skills to novel situationssituations

• evidence for practice in varied evidence for practice in varied contextscontexts

Achievement > Reasoning Achievement > Reasoning AbilityAbility

1.1. OverachievementOverachievement• unusual effort, good instructionunusual effort, good instruction

2.2. Difficulty in applying Difficulty in applying knowledge/skills in unfamiliar knowledge/skills in unfamiliar contextscontexts

• need for integration, cross-course need for integration, cross-course transfertransfer

General issues in General issues in selectionselection

Golden Rules of selectionGolden Rules of selection

Identification criteria must be Identification criteria must be logically and psychologically tied to logically and psychologically tied to the requirements of the day-to-day the requirements of the day-to-day activities that students will pursue.activities that students will pursue. Mathematics?Mathematics? Literary arts?Literary arts? Visual Arts?Visual Arts?

Differentiated selection implies Differentiated selection implies differentiated instructiondifferentiated instruction

Example r = .6 using PR

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Nonverbal Reasoning

Mat

hem

atic

s A

chie

vem

ent

Example of r = .6Example of r = .6


0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Nonverbal Reasoning

Mat

hem

atic

s A

chie

vem

ent

Example r = .6 Example r = .6


0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Nonverbal Reasoning

Mat

hem

atic

s A

chie

vem

ent

Example r = .6Example r = .6


0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Nonverbal Reasoning

Mat

hem

atic

s A

chie

vem

ent

29%71%

Example r = .6Example r = .6

Imprecision of even high Imprecision of even high correlationscorrelations

Given r = .8Given r = .8 What is the likelihood that a student What is the likelihood that a student

who scores in 60-70who scores in 60-70thth PR at Time 1 PR at Time 1 will scores in the 60-70will scores in the 60-70thth PR at Time PR at Time 2?2?

Lohman, D. F. (2003). Lohman, D. F. (2003). Tables of prediction efficienTables of prediction efficienciescies

. .


. .


. .

Proportion of students Proportion of students identified by both testsidentified by both tests

Cut scoreCut score

Correlation between testsCorrelation between tests

0.500.50 0.600.60 0.700.70 0.800.800.90.9

00

Top 1%Top 1% 0.130.13 0.190.19 0.270.27 0.380.38 0.540.54

Top 2%Top 2% 0.170.17 0.230.23 0.310.31 0.420.42 0.580.58

Top 3%Top 3% 0.200.20 0.260.26 0.350.35 0.450.45 0.600.60

Regression to the meanRegression to the mean

The tendency of students with high The tendency of students with high scores to obtain somewhat lower scores to obtain somewhat lower scores upon retestscores upon retest 0 at the mean0 at the mean Increases with distance from the meanIncreases with distance from the mean

Easily predicted from correlationEasily predicted from correlation YYpredpred = Mean + r (Y – mean) = Mean + r (Y – mean)

Causes of Regression to Causes of Regression to the Meanthe Mean

““Errors” of measurementErrors” of measurement Often much larger for high scoring Often much larger for high scoring

studentsstudents Differential growth ratesDifferential growth rates Changes in the abilities measured by Changes in the abilities measured by

the tests at time 1 and time 2 (esp the tests at time 1 and time 2 (esp achievement tests)achievement tests)

Changes in the norming populationChanges in the norming population school sample or national age sampleschool sample or national age sample

Reducing RegressionReducing Regression

Use the most reliable tests available Use the most reliable tests available (judge by SEM on reported score scale)(judge by SEM on reported score scale)

Avoid accepting the highest score as Avoid accepting the highest score as the best estimate of abilitythe best estimate of ability

Average scoresAverage scores Ability and Achievement test scoresAbility and Achievement test scores

Within domain (e.g., math ach & CogAT Q or Within domain (e.g., math ach & CogAT Q or QN)QN)

Achievement at T1 and T2Achievement at T1 and T2 Revolving door policiesRevolving door policies

Combining Combining scoresscores

"And" "Or" "Average"

Test 1 and Test 2 Test 1 or Test 2 Average of Test 1 and Test 2

Figure 5. Plots of the effects of three rules: (a) high scores on test 1 and test 2; (b) high

scores on test 1 or test 2; and (c) high scores on the average of test 1 and test 2.

““And,” “or” or “Average”And,” “or” or “Average”

Screening testsScreening tests

You administer a screening test to You administer a screening test to reduce the number who must be reduce the number who must be administered the admissions testadministered the admissions test

Assume a correlation of r = .6 between Assume a correlation of r = .6 between the two teststhe two tests

Assume students must score at the 95Assume students must score at the 95thth PR or higher on the admissions testPR or higher on the admissions test

What cut score on the screening test What cut score on the screening test will include all of those who would will include all of those who would meet this criterion?meet this criterion?

Proportion of students in top X percent of Proportion of students in top X percent of screening test who exceed the same or a more screening test who exceed the same or a more

stringent cut score on follow up teststringent cut score on follow up test r = .6

Admissions test

Top x % 5% 3% 1%

Scree

nin

g T

est

30% 0.80 0.84 0.91

25% 0.75 0.80 0.87

20% 0.68 0.73 0.82

15% 0.59 0.65 0.75

10% 0.48 0.54 0.65

5% 0.31 0.36 0.48

3% 0.22 0.26 0.36

Proportion of students in top X percent of Proportion of students in top X percent of screening test who exceed the same or a more screening test who exceed the same or a more

stringent cut score on follow up teststringent cut score on follow up test r = .6

Admissions test

Top x % 5% 3% 1%

Scree

nin

g T

est

30% 0.80 0.84 0.91

25% 0.75 0.80 0.87

20% 0.68 0.73 0.82

15% 0.59 0.65 0.75

10% 0.48 0.54 0.65

5% 0.31 0.36 0.48

3% 0.22 0.26 0.36

Screening might make Screening might make sensesense

When admissions test is expensive to When admissions test is expensive to administeradminister

When the correlation between the When the correlation between the admissions & screening test is very admissions & screening test is very highhigh

When there are many more applicants When there are many more applicants than places in the programthan places in the program

When the false rejection rate is not an When the false rejection rate is not an issue issue

Local versus National Local versus National NormsNorms

Except for regional or national talent Except for regional or national talent searches, the PRIMARY reference group is searches, the PRIMARY reference group is not the nation or even the state but the not the nation or even the state but the school or school district.school or school district.

The need for special instruction depends on The need for special instruction depends on the discrepancy between the child’s level of the discrepancy between the child’s level of cognitive and academic development and cognitive and academic development and that of his or her classmates.that of his or her classmates.

Multiple perspectives: Nation, the local Multiple perspectives: Nation, the local population, opportunity-to-learn subgroups population, opportunity-to-learn subgroups within the local populationwithin the local population

Identification of Identification of Talent in Special Talent in Special

PopulationsPopulations

ELL childrenELL children

Identifying academic Identifying academic talenttalent

Not giftednessNot giftedness

Tradeoff Tradeoff

Measuring the right things Measuring the right things approximately for ELL students approximately for ELL students

or or

the wrong things with greater the wrong things with greater accuracyaccuracy

Inference of Aptitude?Inference of Aptitude?

When someone learns in a few When someone learns in a few trials what others learn in many trials what others learn in many trialstrials

Opportunity to learn is criticalOpportunity to learn is critical Common norms appropriate only Common norms appropriate only

if experiences are similarif experiences are similar Placement by achievementPlacement by achievement

Multiple PerspectivesMultiple Perspectives The need for special programming depends The need for special programming depends

most importantly on the discrepancy most importantly on the discrepancy between a child’s achievements & abilities between a child’s achievements & abilities and that of his or her and that of his or her classmatesclassmates

Except for regional talent searches, summer Except for regional talent searches, summer programs that draw from different schools, programs that draw from different schools, etc… Make better use of local norms!etc… Make better use of local norms!

For ELL students in grade 3, compare scores For ELL students in grade 3, compare scores to:to: Other ELL students in grade 3Other ELL students in grade 3 Other students in grade 3 in the district/schoolOther students in grade 3 in the district/school Other grade 3 students in the nationOther grade 3 students in the nation

Multiple Programming Multiple Programming OptionsOptions

Current level of achievement is primary Current level of achievement is primary guideguide

Programming goal: to improve the Programming goal: to improve the achievement at a rate faster than would achievement at a rate faster than would otherwise occurotherwise occur

For on- and below-grade-level For on- and below-grade-level achievement options include: tutors, after-achievement options include: tutors, after-school or weekend classes/clubs, etc. school or weekend classes/clubs, etc. Motivational component critical.Motivational component critical.

For achievement well in advance of peers, For achievement well in advance of peers, consider single-subject accelerationconsider single-subject acceleration

Combining ITBS and Combining ITBS and CogATCogAT

Grades K – 2Grades K – 2 Average CogAT V and ITBS Reading TotalAverage CogAT V and ITBS Reading Total Average CogAT Q and ITBS Math totalAverage CogAT Q and ITBS Math total CogAT NV stands aloneCogAT NV stands alone

Grades 3 – 12Grades 3 – 12 Average CogAT V and ITBS Reading TotalAverage CogAT V and ITBS Reading Total Average CogAT QN and ITBS Math TotalAverage CogAT QN and ITBS Math Total

Use NCE scores – they can be averagedUse NCE scores – they can be averaged Then sort by grade and OTL groupThen sort by grade and OTL group

Integrating ability, Integrating ability, achievement, and teacher achievement, and teacher

ratingsratings See Lohman, D. F. & Renzulli, J. See Lohman, D. F. & Renzulli, J.

(2007). (2007). A simple procedure for combining abA simple procedure for combining ability test scores, achievement test scility test scores, achievement test scores, and teacher ratings to identify ores, and teacher ratings to identify academically talented children. academically talented children.

Teacher Rating (Renzulli Scales) on Teacher Rating (Renzulli Scales) on Learning ability, Motivation, or Learning ability, Motivation, or

Creativity (NOMINATED students only)Creativity (NOMINATED students only)

Verbal Verbal AbilityAbility

Or Or

Quant/NVQuant/NVAbilityAbility

(ALL (ALL Students)Students)


CreativityCreativity


Or Or




Below Avg.Below Avg. Above Avg.Above Avg.

>97>97thth PR PR

>80>80thth PR PR


Or Or





>97>97thth PR PR IIII II

>80>80thth PR PR IVIV IIIIII


Or Or





>97>97thth PR PR IIII IIadmitadmit



Or Or





>97>97thth PR PR IIIIAdmit but Admit but

watchwatch

II



Or Or






>80>80thth PR PR IVIV IIIIIIEnrichmentEnrichment


Or Or






>80>80thth PR PR IVIVTry next yearTry next year

IIIIII


Or Or





>97>97thth PR PR IIIIAdmit but Admit but

watchwatch

IIadmitadmit

>80>80thth PR PR IVIVTry next yearTry next year

IIIIIIEnrichmentEnrichment

Final Thoughts: Using Final Thoughts: Using CogATCogAT

Examine warnings and confidence intervals Examine warnings and confidence intervals on score reportson score reports

Do not screen using Composite scoreDo not screen using Composite score Use V and QN instead (at grade 3 +)Use V and QN instead (at grade 3 +) Combine with Reading Total and Math TotalCombine with Reading Total and Math Total

AverageAverage measures of the same construct; measures of the same construct; Use “or” for measures of different constructsUse “or” for measures of different constructs

To identify talent, measure the right To identify talent, measure the right aptitudes but then compare scores to the aptitudes but then compare scores to the proper norm group(s)proper norm group(s)

Emphasize local norms for in-school Emphasize local norms for in-school programsprograms

ELLELL Compare the performance of the ELL 3Compare the performance of the ELL 3rdrd

grader with that of other ELL 3grader with that of other ELL 3rdrd graders graders Be wary of national norms that you can Be wary of national norms that you can

purchase– esp on nonverbal tests (Raven, purchase– esp on nonverbal tests (Raven, NNAT,…) NNAT,…)

Nonverbal tests have a role to play, but Nonverbal tests have a role to play, but should never stand aloneshould never stand alone

Emphasize the identification of talent Emphasize the identification of talent rather than the identification of rather than the identification of giftednessgiftedness

It is unwise to accept the highest It is unwise to accept the highest score as the best estimate of abilityscore as the best estimate of ability

Combine ability and achievement test Combine ability and achievement test scores in principled ways scores in principled ways

Teacher ratings are only as good as Teacher ratings are only as good as teacher training in making ratingsteacher training in making ratings

Do not simply add teacher ratings and Do not simply add teacher ratings and similar measures to similar measures to ability/achievement scoresability/achievement scores

GeneralGeneral

There is no way to measure innate There is no way to measure innate ability; all abilities are developedability; all abilities are developed

Measures of achievement and ability Measures of achievement and ability differ in degree – not kinddiffer in degree – not kind

Future expertise is built on the base of Future expertise is built on the base of current knowledge in a domain, current knowledge in a domain, reasoning abilities needed for new reasoning abilities needed for new learning in that domain, interest in the learning in that domain, interest in the domain, and the ability to persist in the domain, and the ability to persist in the pursuit of excellence pursuit of excellence

All of which depend on opportunity and All of which depend on opportunity and circumstancecircumstance

The EndThe End

www.cogat.comwww.cogat.com

http://faculty.education.uiowa.edu/http://faculty.education.uiowa.edu/dlohmandlohman

NCE ScoresNCE Scores

Get from the publisher for CogATGet from the publisher for CogAT Table look up (Table 32 in CogAT Table look up (Table 32 in CogAT

Norms Manual)Norms Manual) Convert PR’s to NCE scoresConvert PR’s to NCE scores

In Excel In Excel NCE = NORMINV (PR/100, 50, 21.06)NCE = NORMINV (PR/100, 50, 21.06)

If SAS > 135If SAS > 135 NCE = 21.06 * [(SAS – 100)/16] + 50NCE = 21.06 * [(SAS – 100)/16] + 50

Documents

Using CogAT David Lohman Institute for Research and Policy on Acceleration Belin-Blank Center & Iowa Testing Programs University of Iowa