Distinguishing between Treatment Effects and DIF in a Substance Abuse Outcome Measures Using Multiple Indicator Multiple Causes (MIMIC) Models

Barth B. Riley, Michael L. Dennis

Chestnut Health Systems

Study supported by National Institute on Drug Abuse Grant (NIDA) No. R37 DA11323.

Overview

Differential Item Functioning and Its Impact The Multiple Indicator Multiple Cause (MIMIC)

Model Demographic differences in substance use

and substance abuse treatment Present Study

Differential Item Functioning

DIF: Two groups differ in their likelihood of endorsing an item after controlling for differences on the measured construct.

Group differences in the likelihood of endorsing an item may be due to: Group differences on the latent trait Differential item functioning (DIF) Both

DIF can also occur over time

Differential Item and Test Functioning The presence of DIF items can reduce the

validity of a measure in making between group comparisons.

If DIF is of sufficient magnitude to cause measurement bias against one group relative to another, efforts to interpret outcomes measures becomes complex.

Did the persons change or did the items in the instrument change?

Analysis of DIF

Several approaches have been employed for the analysis of DIF: T tests comparing item parameters between two

groups Mantel-Haenszel contingency tables Logistic regression IRT Likelihood ratio tests

Most of these approaches are limited to comparisons of two groups on a single factor.

Do not directly assess impact of DIF on person measures.

Multiple Indicator Multiple Causes (MIMIC) Models Combines aspects of confirmatory factor

analysis and structural equation modeling. The basic MIMIC models consist of the

following components: A latent variable—the construct being

measured. A set of measured indicators—items Grouping variables such as race and gender

Basic IRT Model

LatentConstruct

LatentConstruct

Item 1Item 1

Item 2Item 2

Item 3Item 3

Item nItem n

……

Indicators

MIMIC Model, No DIF Assumed

LatentConstruct

LatentConstruct

Item 1Item 1

Item 2Item 2

Item 3Item 3

Item nItem n

……

Latent Variable

EthnicityEthnicity

GenderGender

Indirect effects

Indicators

MIMIC Model with DIF Effects

LatentConstruct

LatentConstruct

Item 1Item 1

Item 2Item 2

Item 3Item 3

Item nItem n

……

Latent Variable

EthnicityEthnicity

GenderGender

Effect of DIF is partialed out of the indirect effects Indicators

Direct DIF effect

Study

The purpose of this study was to examine the effect of DIF by time, gender race on the Global Appraisal of Individual Needs (GAIN) Substance Problem Scale

Data were collected from 446 participants as part of a three-year substance abuse early re-intervention study.

Participants (N=446)

Recruited from community-based substance abuse treatment in Chicago in 2004.

Participants were randomly assigned to either outcome monitoring or recovery management checkups, designed to help relapsing participants to return to treatment.

Followed quarterly for 3 years. Participants were predominantly

Male (54.5%) African American (80.2%) Average age: 38.4 years (SD=8.3)

Primary Drug

AlcoholAmphetaminesMariuanaCocaineOpiates

Substance Problem Scale

The Substance Problem Scale (SPS) measures problems with alcohol/drug use during the past month, including abuse, dependence and substance-abuse health problems.

Consists of 16 dichotomous items Based on DSM-IV-TR criteria for substance

abuse and substance dependence. Internal consistency: .9 Test-retest reliability: .73

Model

In order to assess treatment effects over time, a multilevel framework was used: Level 1: Time: random effect Level 2: Person: fixed effects

Treatment variables: Random assignment to recovery management Days in outpatient, intensive outpatient and residential

treatment DIF factors: gender and ethnicity One and two parameter IRT models were compared.

MIMIC Model: Within Level

SPSSPS

TimeTime

TxParticipation

TxParticipation

SPS 1SPS 1

SPS 2SPS 2

SPS 3SPS 3

SPS nSPS n

……

Control for DIF

MIMIC Model: Between Level

SPSSPS

RaceRace

GenderGender

RMCRMC

SPS 1SPS 1

SPS 2SPS 2

SPS 3SPS 3

SPS nSPS n

……

Control for DIF

OpiatesOpiates

Goodness of Fit

Model CFI TLI RMSEA

1 Parameter IRT

Basic IRT model .974 .970 .099

MIMIC No DIF .972 .961 .076

MIMIC DIF .974 .970 .075

2 Parameter IRT

Basic IRT model .952 .994 .045

MIMIC No DIF .975 .994 .03

MIMIC DIF .976 .994 .03

N Cases = 400, N Observations = 5393

Time DIF

-0.75

-0.5

-0.25

0

0.25

0.5

0.75

1

1.25

1.5

1.75

0 3 6 9 12 15 18 21 24 27 30 33 36

Time (Months)

Ite

m D

iffi

cu

lty

(b)

Hiding Use

Use Weekly

Using/Responsibilities not met

Need more AOD get same effect

Made you depressed

Used more often

Time DIF

Unable to cut down use

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

0 3 6 9 12 15 18 21 24 27 30 33 36

Time (Months)

Item

Dif

ficu

lty

(b)

Gender DIF

0

0.5

1

1.5

2

2.5

Caused Health Problems Made situation unsafe

MaleFemale

Ethnicity DIF

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Causing fights, trouble Widthdrawal symptoms

HispanicNon-Hispanic

Primary Drug DIF

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Made situation unsafe Withdrawal symptoms

OpiatesOther Drugs

Group Differences on DIF Factors

Factor

No DIF Model DIF Model

Z Sig. Z Sig.

Time -0.271 .001 -0.281 .001

Gender 0.217 .041 0.206 ns

Ethnicity -0.733 ns -0.845 ns

Primary Drug 0.333 .015 0.257 ns

Treatment Effects

Treatment Variables

No DIF DIF

Z Sig. Z Sig.

Any outpatient tx. -0.606 .001 -0.605 .001

Times in outpatient tx. 0.003 ns 0.003 ns

Any intensive outpatient -0.325 ns -0.324 ns

Days, intensive outpatient tx. -0.007 ns -0.007 ns

Any residential treatment 0.435 .001 0.434 .001

Nights in residential tx. -0.306 .001 -0.306 .001

Any methadone tx. 0.636 .001 0.635 .001

Days taking methadone -0.023 ns -0.023 ns

Recovery Monitor. Checkups -0.248 .018 -0.249 .018

Conclusions

The MIMIC model is a promising tool for assessing the presence and impact of DIF on at the scale level (DTF).

Controlling for DIF reduced differences in SPS measures as a function of gender and primary drug.

Treatment effects as measured by the SPS were not affected by gender, ethnicity, primary drug or time DIF.

MIMIC: Strengths

Assess DIF and DTF on multiple factors DIF factors can be discrete or continuous

variables Distinguish between treatment and DIF

effects Can be used in conjunction with longitudinal

analysis methods (e.g., multilevel modeling).

MIMIC: Limitations/Caveats

In order to specify the model, at least one item must be free of DIF (or have minimal DIF).

Can not detect non-uniform DIF—DIF in the discrimination parameter

Obtaining group specific item parameters is not straightforward

Assumes consistent factor structure across groups

Useful References

Muthén, B. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557-585.

Fleishman, L.A., & Lawrence, W.F. (2003). Demographic variation in SF-12 scores: True differences or differential item functioning? Medical Care, 41(7 Suppl.) III75-III86.

MacIntosh, R. & Hashim, S. (2003). Converting MIMIC model parameters to IRT parameters in DIF analysis. Applied Psychological Measurement, 27, 372-379.

Finch, H. (2005). The MIMIC Model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4):278-295.

Contact Information

A copy of this presentation will be at: www.chestnut.org/li/posters

For information on this method and a paper on it, please contact Barth Riley at bbriley@uic.edu.