Download ppt - A Practitioner’s Introduction to Equating

A Practitioner’s Introduction to Equating

Joseph Ryan, Arizona State UniversityFrank Brockmann, Center Point Assessment Solutions

with Primers on Classical Test Theory (CTT) and Item Response Theory (IRT)

Assessment, Research and Evaluation Colloquium Neag School of Education, University of Connecticut October 22, 2010

Workshop:

Acknowledgments• Council of Chief State School Officers (CCSSO)

• Technical Issues in Large Scale Assessment (TILSA) and Subcommittee on Equating, part of the State Collaborative on Assessment and Student Standards (SCASS)

• Doug Rindone and Duncan MacQuarrie, CCSSO TILSA Co-Advisers Phoebe Winter, Consultant Michael Muenks, TILSA Equating Subcommittee Chair

• Technical Special Interest Group of National Assessment of Educational Progress (NAEP) coordinators

• Hariharan Swaminathan, University of Connecticut

• Special thanks to Michael Kolen, University of Iowa

Workshop Topics

The workshop covers the following topics:

1. Overview - Key concepts of assessment, linking, and equating

2. Measurement Primer – Classical and IRT theories

3. Equating Basics

4. The Mechanics of Equating

5. Equating Issues

1. Overview

Key Concepts inAssessment, Linking, Equating

Assessment, Linking, and Equating

Validity is…… an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.

(Messick, 1989, p. 13)

Validity is the essential motivation for developing and evaluating appropriate linking and equating procedures.

Assessment, Linking, and EquatingThe Linking Continuum

Equating

Scores are matched or paired

Scores do NOT have thesame meaning or interpretation

Scores arematched or paired

Scores have the SAMEmeaning or interpretation

2007NRT

Gr 5 Test

2007NRT

Gr 5 Test

2007SBA

Gr 5 Test

2007SBA

Gr 5 Test

2006SBA

Gr 5 test

2006SBA

Gr 5 test

2007SBA

Gr 5 Test

2007SBA

Gr 5 Test

(strongest link)(weaker kinds

of linking)

Linking and Equating

• Equating• Scale aligning• Predicting/Projecting

Holland in Dorans, Pommerich and Holland (2007)

Misconceptions About Equating

Equating is…

• …a threat to measuring gains.

• …a tool for universal applications.

• …a repair shop.

• …a semantic misappropriation.

MYTH

WISHFUL THOUGHT

MISCONCEPTION

MISUNDERSTANDING

2. Measurement Primer

Classical Test Theory (CTT) Item Response Theory (IRT)

Classical Test Theory

The Basic Model

Oobserved

score

Ttrue score

Eerror

(with some MAJOR assumptions)

= +• Reliability is derived from the ratio of error score to true score • Key item features include:

Difficulty Discrimination Distractor Analysis

Reliability reflects the consistency of students' scores• Over time, test retest• Over forms, alternate form• Within forms, internal consistency

Validity reflects the degree to which scores assess what the test is designed to measure in terms of

• Content• Criterion related measures• Construct

Classical Test Theory

Item Response Theory (IRT)

The Concept

An approach to item and test analysis that estimates students’ probable responses to test questions, based on

• the ability of the students• one or more characteristics of the test items


• IRT is now used in most large-scale assessment programs

• IRT models apply to items that use dichotomous scoring with right (1) or wrong (0)

answers and polytomous scoring with items scored with ordered

categories (1, 2, 3, 4) common with written essays and open-ended constructed response items

• IRT is used in addition to procedures from CTT

INFO


IRT Models

All IRT models reflect the ability of students. In addition, the most common basic IRT models include:

The 1-parameter model – (aka Rasch model) models item difficulty

The 2-parameter model – models item difficulty and discrimination

The 3-parameter model – models item difficulty, discrimination and pseudo guessing


IRT Assumptions

Item Response Theory requires major assumptions:

• Unidimensionality

• Item Independence

• Data-Model Fit

• Fixed but arbitrary scale origin


A Simple Conceptualization

0 1 2 3-3 -2 -1

Easier Items

Harder Items

Higher Ability

Students

Lower Ability

Students

Alex

Blake

Chris

Devon

Item 1 Item 2 Item 3

-1.5 +2.25

Item Response Theory (IRT)Probability of a Student Answer

Item Response Theory (IRT)Item Characteristic Curve for Item 2


IRT and Flexibility

IRT provides considerable flexibility in terms of

• constructing alternate tests forms• administering tests well matched or adapted to

students’ ability level• building sets of connected tests that span a

wide range (perhaps two or more grades)• inserting or embedding new items into existing

test forms for field testing purposes so new items can be placed on the measurement scale

INFO

3. Equating Basics

Basic Terms (Sets 1, 2, and 3)

Equating Designs (a, b, c)

Item Banking (a, b, c, d)

Basic Terms Set 1

Column A Column B __Anchor Items A. Sleepwear__Appended Items B. Nautically themed__Embedded Items apparel

C. Vestigial organs D. EMIP learning

module

USEFUL TERMS

Basic Terms Set 2

Pre-equating -

Post equating -

USEFUL TERMS

For each term, make some notes on your handout:

Basic Terms Set 3

Horizontal Equating –

Vertical Equating (Vertical Scaling) –

Form-to-Form (Chained) Equating – Item Banking –

USEFUL TERMS

For each term, make some notes on your handout:

Equating Designs

a. Random Equivalent Groupsb. Single Groupc. Anchor Items

Equating Designs

a. Random Equivalent Groups

Testing Population

RandomSample Group 2

RandomSample Group 1

Test Form A

Test Form B

Equating Designs

b. Single Group

Testing PopulationRandom

Sample Group

Test Form A (first)

Test Form B (second)

CAUTION

The potential for order effects is significant--equating designs that use this data collection method should always be counterbalanced!

Equating Designs

b. Single Group with Counterbalance

Testing Population orTested Sample

RandomSubgroup 2

RandomSubgroup 1

Test Form A (first)

Test Form B (second)

Test Form B (first)

Test Form A (second)

Equating Designs

c. Anchor Item Design

common items

Testing Sample 1

Test Form A Items

Anchor Items

Test Form B Items

Anchor Items

Testing Sample 2

not always at the end

Equating Designs

c. Anchor Item Set

Anchor Selection

GRADE 5Mathematics Test Form A

(50 test i tems)

Content Standard 110 items










GRADE 5Mathematics Anchors Set

(10 items)PROPER

c. Anchor Item Designs

• Internal/Embedded

• Internal/Appended

• External

USEFUL TERMS

Equating Designs

Equating DesignsInternal Embedded Anchor Items

Item 1 (B)Item 2 (Anchor)Item 3 (B)Item 4 (B)Item 5 (Anchor)Item 6 (B)Item 7 (B)Item 8 (B)Item 9 (Anchor)Item 10 (B)Item 11 (B)Item 12 (Anchor)Item 13 (B)Item 14 (B)Item 15 (Anchor)

Item 1 (B)Item 2 (Anchor)Item 3 (B)Item 4 (B)Item 5 (Anchor)Item 6 (B)Item 7 (B)Item 8 (B)Item 9 (Anchor)Item 10 (B)Item 11 (B)Item 12 (Anchor)Item 13 (B)Item 14 (B)Item 15 (Anchor)

Item 1 (A)Item 2 (Anchor)Item 3 (A)Item 4 (A)Item 5 (Anchor)Item 6 (A)Item 7 (A)Item 8 (A)Item 9 (Anchor)Item 10 (A)Item 11 (A)Item 12 (Anchor)Item 13 (A)Item 14 (A)Item 15 (Anchor)

Item 1 (A)Item 2 (Anchor)Item 3 (A)Item 4 (A)Item 5 (Anchor)Item 6 (A)Item 7 (A)Item 8 (A)Item 9 (Anchor)Item 10 (A)Item 11 (A)Item 12 (Anchor)Item 13 (A)Item 14 (A)Item 15 (Anchor)

Test Form A Test Form B

Item 2 (Anchor)Item 5 (Anchor)Item 9 (Anchor)Item 12 (AnchorItem 15 (Anchor)

Embedded, InternalAnchor Items

Equating DesignsInternal Appended Anchor Items

Item 1 (B)Item 2 (B)Item 3 (B)Item 4 (B)Item 5 (B)Item 6 (B)Item 7 (B)Item 8 (B)Item 9 (B)

Item 10 (B)


Item 10 (B)

Item 1 (A)Item 2 (A)Item 3 (A)Item 4 (A)Item 5 (A)Item 6 (A)Item 7 (A)Item 8 (A)Item 9 (A)

Item 10 (A)


Item 10 (A)


Item 11 (C)Item 12 (C)Item 13 (C)Item 14 (C)Item 55 (C)




Form

-Spe

cific

Item

s

Form

-Spe

cific

Item

s

Anc

hor

Item

s

Anc

hor

Item

s

Appended, InternalAnchor Items

Equating DesignsExternal Anchor Items


Item 10 (B)


Item 10 (B)


Item 10 (A)


Item 10 (A)






Par

t 1

Par

t 1

Par

t 2

Par

t 2

Appended, ExternalAnchor Items

Equating Designs

Guidelines for Anchor Items

• Mini-Test

• Similar Location

• No Alterations

• Item Format Representation

RULES of

THUMB

3. Equating Basics

Basic Terms (Sets 1, 2, and 3)

Equating Designs (a, b, c)

Item Banking (a, b, c, d)

Item Banking

a. Basic Conceptsb. Anchor-item Based Field Testc. Matrix Samplingd. Spiraling Forms

• An item bank is a large collection of calibrated and scaled test items representing the full range, depth, and detail of the content standards

• Item Bank development is supported by field testing a large number of items, often with one or more anchor item sets.

• Item banks are designed to provide a pool of items from which equivalent test forms can be built.

• Pre-equated forms are based on a large and stable item bank.

Item Bankinga. Basic Concepts

b. Anchor Item Based Field Test Design

RULE of THUMB

Field test items are most appropriately embedded within, not appended to, the common items.

Item Banking

Item Banking

• Items can be assembled into relatively small blocks (or sets) of items.

• A small number of blocks can be assigned to each test form to reduce test length.

• Blocks may be assigned to multi forms to enhance equating.

• Blocks need not be assigned to multi forms if randomly equivalent groups are used.

c. Matrix Sampling

Item Bankingc. Matrix Sampling

Tests forms can be assigned to individual students, or students grouped in classrooms, schools, districts, or some other units.

1. “Spiraling” at the student level involves assigning different forms to different students within a classroom.

2. “Spiraling” at the classroom level involves assigning different forms to different classrooms within a school.

3. “Spiraling” at the school or district level follows a similar pattern.

Item Bankingd. Spiraling Forms

d. Spiraling Forms

Item Banking

Item Bankingd. Spiraling Forms

Spiraling at the student level is technically desirable:• provides randomly equivalent groups• minimizes classroom effect on IRT estimates

(most IRT procedures assume independent responses)

Spiraling at the student level is logistically problematic:

• exposes all items in one location• requires careful monitoring of test packets and

distribution• requires matching test form to answer key at the

student level

It’s Never Simple!

Linking and equating procedures are employed in the broader context of educational measurement which includes, at least, the following sources of random variation (statistical error variance) or imprecision.

• Content and process representation • Errors of measurement• Sampling errors• Violations of assumptions• Parameter estimation variance• Equating estimation variance

IMPORTANT

CAUTION

4. The Mechanics of Equating

The Linking-Equating Continuum Classical Test Theory (CTT) Approaches Item Response Theory (IRT) Approaches

The Linking-Equating Continuum

Linking is the broadest terms used to refer to a collection of procedures through which performance on one assessment is associated or paired with performance on a second assessment. Equating is the strongest claim made about the relationship between performance on two assessments and asserts that the scores that are equated have the same substantive meaning.

USEFUL TERMS


different forms of linking

equating (strongest kind of linking)

The Linking-Equating ContinuumFrameworks


Moderation Equating

Scores do NOT have the same meaning or interpretation

Scores have the SAMEmeaning

or interpretation

Linking Procedures/Approaches

CTT IRTLinear

Equipercentile

Common Item

Pre- and Post-equating

Calibration

Projection

Common PersonPool/Item Bank Development

In 1992, Mislevy described four typologies of linking test forms: moderation, projection, calibration, and equating (Mislevy, 1992, pp. 21-26). In his model, moderation is the weakest form of linking tests, while equating is considered the strongest type. Thus, equating is done to make scores as interchangeable as possible.

The Linking-Equating ContinuumEquating – strongest form of linking, invariant across populations, maintains substantive meaning Calibration – may use equating procedures, not necessarily invariant across populations, and substantive meaning might not be preserved Prediction/Projection – unidirectional statistical procedure for predicting scores or projecting distributions Moderation – weakest form of linking, may be statistical or judgmental (social), based on comparisons of distributions or panel/reviewers decisions.

USEFUL TERMS

CTT Linking-Equating Approaches

a. Mean Methodb. Linear Methodc. Equipercentile Method

CTT Linking-Equating Approachesa. Mean Method

• Adjusts one set of scores based on the difference in the means of two tests

• Assumes a constant difference in the scales across all scores

• Useful for carefully developed and parallel or close-to-parallel forms

• Simple, but strains assumptions of parallel forms


b. Linear Method

• Based on setting standardized deviation scores from two tests equal

• Can be done in raw score scale with simple linear regression

CTT Linking-Equating Approachesb. Linear Method

Linear Equating

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

Raw Score on Form B

Ra

w S

co

re o

n F

orm

A


c. Equipercentile Method

• Based on scores that correspond to the same percentile rank position from two tests

• Does not assume a linear relationship between the two tests

• Provides for linking scores across the full range of possible test scores

• May require “smoothing” of the distributions, especially with small samples

CTT Linking-Equating Approachesc. Equipercentile Method

a. common itemsb. common people or randomly

equivalent groups treated as being the same people

IRT Linking-Equating Approaches


IRT linking and equating approaches:

• provide flexibility and are applicable to many settings

• provide consistency by employing the IRT model being used for calibration and scaling

• provide indices that reveal departures from what is expected (tests of fit)


a. Common Items

Approaches can be based on:

1. Applying an equating constant

2. Estimating item parameters with fixed or concurrent/simultaneous calibration

3. Applying the Test Characteristic Curve procedure (TCC) of Stocking & Lord, 1983


a. Common Items

Applying an equating constant

• Appropriate when two or more tests have a common set of anchor items and also some items unique to each form

• Requires selecting one form or some other location on the scale as the origin of the scale

IRT Linking-Equating Approachesa. Common Items

1. Applying an Equating Constant

0 1 2 3-3 -2 -1

Easier Items

Harder Items

Higher Abil i ty

Students

Lower Abil i ty

Students

Item A

Item B

Item CTest Form X(20 items) = test item

A B C

0



0 1 2 3-3 -2 -1

Easier Items

Harder Items

Higher Abil i ty

Students

Lower Abil i ty

Students

Item CTest Form Y(20 items) = test i tem

Item B

A B C

Item A

0


Common Item Approach

Applying an equating constant

0 1 2 3-3 -2 -1

Easier Items

Item C

2(0+2)

3(1+2)

4(2+2)

5(3+2)

-1(-3+2)

0(-2+2)

1

Harder Items

Test Form X (top)vs.

Test Form Y (bottom)A B C

Item B

A B C

Item A

THREE ITEMS IS NEVER -- EVEN UNDER MASSIVE DELUSIONARY INFLUENCES -- ENOUGH ITEMS!

EVEN WITH 15 TO 20 ITEMS -- A MINIMUM -- IT NEVER WORKS THIS SIMPLY.

CAUTION



0 1 2 3-3 -2 -1

A B C

Item CItem A

Item B

4 5

Easier Items

Harder Items

Test Form X* (adjusted)and

Test Form Y

Form Y Form X (Y-X)

Item A 0.5 -1.5 2

Item B 1.0 -1.0 2

Item C 1.5 -0.5 2

Sum 3.0 -3.0 6

Average 1.0 -1.0 2

Constant = Form Y - Form X = 2

If C= Y – X ; then Y = X + C

a. Common Items1. Determining an Equating Constant


CLOSER LOOK


a. Common Items1. Applying an Equating Constant

• The common items used for equating are the anchor

items

• Generally 15 to 20 items are needed for common item equating

• Not all items designed as anchor items will work effectively

• The anchor items should be in the same location on the tests

• The anchor items should reflect the content, format and difficulty range of the whole test


Test Form

XTest Form

Y1. Designate this as the Base form, which

defines the scale origin.2. Calibrate parameters (difficulty,

discrimination, and guessing) of all items. 3. Treat the item parameters of the anchor

items as fixed.

4. Use parameters of the anchor items from Form X for the same items (anchors) on Form Y.

Anchor Items

1, 5, 7, 10Etc.

5. Calibrate the Form Y items using the fixed parameter values of the anchor items.

6. Treat all other items and their parameters as free to vary

The resultant calibration of Form Y will be on the same scale as Form X ; it is “anchored” through the fixed values of the common items .


2. Fixed Calibration

Consider the following:

• 500 students take Form X

• 500 students take Form Y

• 1,000 take the anchor items

• The data for all students are stacked as shown on the next slide…

Test Form

XTest Form

Y15 Anchor

Items

40 Items 40 Items


2. Concurrent or Simultaneous Calibration

• Data are calibrated on 1,000 students• Students each “take” 65 items• Students are missing data on the form they did not take. • All students respond to the anchor items.

Form X Items (25 items)

Anchor Items(15 Items)

Form Y Items(25 Items)

The 500 students who take Form X will take 40 items

Item Responses to 25 items


Missing Data

The 500 students who take Form Y will take 40 items

Missing Data



500 students 1,000 students 500 students


2. Concurrent or Simultaneous Calibration

• Developed by Stocking and Lord (1983)

• Very flexible and widely used

• Commonly applied with the 2- and 3- parameter IRT models.


3. Test Characteristic Curve (TCC) Procedures

• IRT scales have an arbitrary origin and an arbitrary scale

spacing e.g., size of each unit of measurement.

Origin is selected and fixed Scale spread is expended or reduced

• Item parameter estimates for the same items from two independent calibration will differ due to

Origin and scale differences Characteristics of other items Possibly sampling and estimation error



• If two scales differ in origin (location) and spread (variability), a linear transformation can be applied to one scale to re-express or transform it to be on the other scale

• The choice of what scale to use is informed by considering the intended use of the items, test forms, or item bank

• The figures on the next slide illustrate the basic idea of the TCC method



a. Common Items


• Transforms the Item parameter values for the common items

on one test form to be on the same scale as their corresponding parameter values on the other (target) form

• Requires two constants: the parameters are multiplied by one constant and then added to the second constant

• Begins with carefully chosen initial values for the constants

• Refines the constants to minimize the differences in estimated scores based on the transformed test form and the target form

• Never as simple as the theory




The same students, or two groups sampled to be equivalent on critical relevant characteristics, take Form X and Form Y; the forms do not have any common items

• Example: students’ average ability on Form X is -1.0 (low ability) and their average ability on Form Y is +1.0 (high ability)

• Differences in students’ abilities cannot explain the differences in the performance on Forms X and Y since the same students (common students) take both forms

IRT Linking-Equating Approachesb. Common Persons

or Random Equivalent Groups

QUESTION

How can the same group of students have two different mean abilities?

• The difference in mean performance reflects the difference in the difficulty of the two forms

• The test forms must be different in difficulty since the students’ abilities were held constant (same students)

• On Form X, students look less able with a mean of -1; on Form Y students look more able with a mean of +1

• Form X is harder than Form Y in that it makes students look less able; the test forms differ by +2 units

• The difference of +2 is used as a linking constant to adjust the tests onto a single scale in the same way as a linking constant derived from common items

IRT Linking-Equating Approachesb. Common Persons

or Random Equivalent Groups

5. Equating Issues Substantive Concerns Technical Issues Quality Control Issues

• Test design, development & administration• Scoring, analysis and equating

Technical Documentation Accountability Compliance Item Formats and Platforms

Common Equating Concerns/Issues

Substantive Concerns

• Validity is the central issue

• Validity evidence must document fairness, absence of bias, and equal access for all students

• Carefully planned and rigorously monitored item and test form development are the most essential ingredients for successful equating

• Equating goes bad through items and test forms, not in the psychometrics


Technical Issues

• Examining and testing IRT

assumptions

• Conducting and documenting IRT tests of fit Data to model fit Linking/equating fit

• Item Parameter Drift

Common Equating Concerns/IssuesQuality Control Issues

Test design, development, and administration problems

• Changes in content standards or test specifications• Item contexts that differ between forms and affect

performance on anchor items• Anchor items that appear in very different locations among

forms• Item misprints/errors• Unintended accommodations (maps or periodic tables on

walls, calculators, etc.)• All manner of weird and unimaginable stuff and happenings

Common Equating Concerns/IssuesQuality Control Issues

Item scoring, analysis, and equating quality issues

• Non-standard scoring criteria or changes in scoring procedures

• Redefinition in scoring rubrics, variation in benchmark papers• Item parameter drift• Departures from specified equating procedures• Unreliable and/or inconsistent item performance or score

distributions• Departures from specified data processing protocols• All manner of weird and unimaginable stuff and happenings


Technical Documentation • General technical reports• Standards setting reports• Equating technical reports• Specify requirements for documentation

in RFPs, with TAC reviews and due dates

QUESTION

Can an independent contractor replicate the equating results?


Accountability Concerns

• Standard Setting

• Adequate Yearly Progress (AYP)


Item Formats and Platforms

• Open-ended or Constructed

Response Tasks

• Writing Assessment

• Paper-and-Pencil and Computerized Assessments

References

Dorans, N. J., Pommerich, M., & Holland, P. W. (2007). Linking and aligning scores and scales. Statistics for social and behavioral sciences. New York: Springer.

Linn, R.L. (1993) Linking results of distinct assessments. Applied Measurement in Education, 6, 83-102.

Mislevy, R.J. (1992) Linking educational assessments: Concepts, issues, methods, and prospects. Princeton, NJ: Educational Testing Service

Ryan, J. and Brockmann, F. (2009). A Practitioner’s Introduction to Equating with Primers on Classical Test Theory and Item Response Theory. Washington, D.C.:Council of Chief State School Officers (CCSSO).

A Practitioner’s Introduction to Equating

Joseph Ryan, Arizona State [email protected]

Frank Brockmann, Center Point Assessment [email protected]

END

mailto:[email protected]

mailto:[email protected]