Reflections and illustrations on DIF

Paul De BoeckK.U.Leuven

25th IRT workshop Twente, October 2009

Reflections and illustrations on DIF

Paul De BoeckUniversity of Amsterdam

25th IRT workshop Twente, October 2009

Is DIF a dead topic?A non-explanatory approach

Paul De BoeckUniversity of Amsterdam

25th IRT workshop Twente, October 2009

Is there life after death for DIF?A non-explanatory approach

Paul De BoeckK.U.Leuven

25th IRT workshop Twente, October 2009

The three DIF generations

Zumbo, Language Assessment Quarterly, 2007 1st generation:

from “item bias” to “differential item functioning”2nd generation:

modeling item responses, IRT, multidimensional models

3rd generation:explanation of DIF

The end of history“ .. the pronouncements I hear from some quarters that

psychometric and statistical research on DIF is dead or near dying ..”

Outline

• Issues

• Reflections and more

• Possible answers

Issues

• Anchoring

• Statistic

• Indeterminacies

I apologize, ..

• There are already so many methodsyes

• The best among the existing methodsare very good methodsyes

• They are standard and good practiceyes

• Do we really need more?no, therefore no real issues

• And still

1. Anchoring

• Blind, iterativePurification- all other in step 1- nonrejected items in following steps

• A priori set, testThey work

based on pragmatism and a heuristic,on prior theory, what can one want more?

2. Statistic and its distribution

Based on difference per item or set of items• MH statistic• ST-p-DIF • Bu from SIBTEST• LR test statistic• Raju distanceOther• Parameter estimatesThey work, what can one want more?

3. Indeterminacies with an IRT modeling approach

Basic model is

• 1PL or Rasch modelfor uniform DIF

• 2PLfor uniform and non-uniform DIF type 1

• 2PL multidimensionalfor uniform and non-uniform DIF type 2

• Difficulties – uniform DIFAdditive or translational indeterminacyβfi = βri + δβi β*fi = βri + δ*βi δ*βi = δβi + cβ

γ* = γ – cβ

βfi , βri focal group and reference group difficulties δβi DIF effect γ group effect * transformed values

Invariance of DIF explanation

• δβi = Σk=0ωkXik (+ εi)

Xik: value of item i on item covariate k ωk: weight of covariate k in explaining

DIF k=0 for intercept

• ωk>0 are translation invariant, and only these covariates have explanatory value

• Degrees of discriminationnon-uniform DIF type 1Multiplicative indeterminacy αfi = αri x δαi

α*fi = αri x δ*αi

δ*αi = δαi x cα

σθf* = σθf / cα

additive formulation discrimination DIF

• Loadings for multidimensional models

The indeterminacies look a little embarrassing, because the results depend on one’s choice.

Reflections

• Random item effects

• Item mixture models

• Robust statistics

Intro: Beliefs

• DIF is gradualwhy not a random item effect?

• DIF or no DIFwhy not a latent class of DIF items?

• DIF items are a minoritywhy not identify outliers?

-4 -2 0 2 4

-4-2

02

4

Reference

Fo

ca

la

Where is the DIF?

-4 -2 0 2 4

-4-2

02

4

Reference

Fo

ca

lb

Where is the DIF?

-4 -2 0 2 4

-4-2

02

4

Reference

Fo

ca

lc

Where is the DIF?

-4 -2 0 2 4

-4-2

02

4

Reference

Fo

ca

ld

Where is the DIF?

Intro: ANOVA approach

• ηgpi = ln(Pr(Ygpi=1)/Pr(Ygpi=0))• ηgpi =

μ overall mean+ λgp = αθgp person effect, ability θgp ~ N (0,1)+ λi = βi item effect, overall item difficulty+ λg = γg group effect+ λgp interaction p x g does not exist+ λgpi = α’iθgp interaction pwg x i

+ λgi = β’gi interaction i x g uniform DIF

+ λgpi = α’’giθgp interaction pwg x i x g non-uniform DIF type 1

2PL version

• + λgpi = α’iθgp

+ λgpi = α’’giθgp interaction pwg x i x g is non-uniform DIF Type

1

• + λgpi = α’iθgp1

+ λgpi = α’’giθgp2 interaction pwg x i x g is non-uniform DIF Type

2

Secondary dimension DIF

• ηgpi = (αi + gδαi)θgp + (βi + gδβi) + λg = αiθgp + gδαiθgp + (βi + gδβi) + λg

• Secondary-dimension DIF ηgpi = αiθgp1 + gδαiθgp2 + (βi + gδβi) + λg

Cho, De Boeck & Wilson, NCME 2009

g = 0 reference groupg = 1 focal group

• can explain uniform DIF

ηgpi = αiθgp1 + gδαiθgp2 + (βi + gδβi) + λg

gδαiμθg2 + gδαiθ’gp2 = gδβi

Cho, De Boeck & Wilson, NCME 2009

• Different from the MIMIC model

• Secondary dimension DIF

ηgpi

θgp1

θgp2

G

ηgpi

θgp1

gθgp2

G

1. Random item effects

• Within group random item effects(βri, βfi) ~ N(μβr, 0, σ2

βr, σ2βf, ρβrβf)

(βi, βf-gi) ~ N(μβr, 0, σ2β, σ2

βf-g, ρββf-g)small number of parameters²

• Idea based on Longford et al in Holland and Wainer (1993) for the MHthere is evidence that the true DIF parameters are distributed continuouslyVan den Noortgate & De Boeck, JEBS, 2005Gonzalez, De Boeck & Tuerlinckx, Psychological Methods, 2008De Boeck, Psychometrika, 2008

2. Latent class of DIF items

• Asymmetric DIF is exported to other items

• Is avoided when DIF items are removed, appropriate removing eliminates interaction

• Basis of purification process

• Let us make a latent class for items to be removed, and identify the DIF items on the basis of their posterior probability

Item mixture model

• ηgpi|ci=0 = θgp + βi non-DIF classηgpi|ci=1 = θgp + βgi DIF class

θrp + βi θrp + β0i

θfp + βi θfp + β1i

non-DIF DIF

reference

focal Frederickx, Tuerlinckx, De Boeck & Magis, resubmitted 2009

• further model specifications:- item effects are random- normal for the non-DIF items- bivariate normal for the DIF item difficulties- group specific normals for abilities

• Simulation study 1PL

P=500, 1000 2 I = 20, 50 x 2#DIF = 0, 5 (1.5, 1, 0.5, -1, -1.5) x 2 μθ1 = 0, μθ2 = 0, 0.5, x 2 = 16 μβ = μβ0 = μβ1, σ2

β = σ2β0 = σ2

β1 = 1, ρβ0β1 = 0five replicationsMCMC WinBUGS prior β variance: Inv Gamma, Half normal, Uniformdistributional parameters are estimatedposterior prob determines whether flagged as DIF

• Results simulation studyaverage #errorsLRT 1.64MH 1.39ST-p-DIF 0.65mixture inverse gamma 0.30mixture normal 0.36mixture uniform 0.40item mixture does better or equally good then every other traditional method in all 16 cells

• More results- results of mixture model are not affected by DIF being asymetrical- neither by true distribution of item difficulties (normal vs uniform)

3. DIF items are outliers

• Outlying with respect to the item difficulty difference between reference and focal group

• Types of difference:- simple difference- standardized – divided by standard error- Raju distance – first equal mean difficulty linking, then standardizeτi = I/(I-1)2 x (di - d.)2/s2

d

is beta (0.5, (I-2)/2) distributed if d i is normally distributed

• Go robust:d. is replaced by the mediansd is replaced by mean absolute deviation

Taking advantage of the fact that interitem variation is an approximation of se if robustly estimatedDe Boeck, Psychometrika 2008Magis & De Boeck, 2009, rejected

20 items, nrs 19 and 20 are the true DIF items

Simple difference

Simple difference

Standardized difference

Standardized difference

Raju

Raju

• Simulation study 1PLP=500, 1000 2 I = 20, 40 x 2%DIF = 0%, 10%, 20% x 3 size of DIF = 0.2, 0.4, 0.6, 0.8, 1.0 x 5μθ1 = 0, μθ2 = 0, 1 x 2 = 120100 replications

• Results

0% DIFMHSIBTESTLogisticRaju classicRaju robust

Type 1 errors ≈ 5%

• Results DIF size = 1, P=1000, I=40, equal μθ

10% DIF 20%DIF

Type 1 Power Type 1 Power

MH 0.10 1.00 0.23 1.00SIBTEST 0.10 0.98 0.21 0.97Logistic 0.10 1.00 0.20 1.00Raju classic 0.00 0.93 0.00 0.41Raju robust 0.04 1.00 0.02 1.00

• Results are similar for unequal mean abilities

• Results are similar but less pronouncedfor smaller P and smaller DIF size

Possible answers

• Anchoring?Anchor set memberschip is binary latent item variable, or, the clean set of items

• Statistic?Robust statisticworks also for nonparametric approaches

• Indeterminacy?(go explanatory)no issue for random item model, look at the covequal means in item mixture approachequal means for Raju distance

• Item mixtures and robust statistics do in one step what purification does in several steps, item by item, and through different purification steps – purification is approximate:

• They both give a rationale for the solving the indeterminacy issue

• Random item effect approach is not sensitive to indetermincay

• Si no è utile è ben ispirazione

Good for other purposes or a broader concept than DIF, for qualitative differences between groups

• Random item models

• Item mixture models

• Robust statistics IRT

Thank you, and stay alive

• Dimensionalityuniform DIF and non-uniform DIF type 2

321

56

4

7 7

654

321

3 2 1

7654