How ________ is your thermometer?

Good measure is central to ability to assess

well

Classical Test Theory

Generalizability Theory

Item Response Theory

=

Variance discriminate, but variance due to what (facets)?

Decompose variance into e.g. person, p

and item, i. Identify facets causing desired,undesired variation

•Probabilistic rather than deterministic•Item (question) level discrimination, not just test level

•Account for person ability and item difficulty in same scale

A person should have greater probability of

success on easier item than more difficult

item.

A more able person should have greater

probability of success on assessment items

than a less able person.

Rasch analysis (in a nutshell)-how good is

your test item fitting to these assertions?

(Person ability – Item Difficulty)

Bar height = 1.80m

(item difficulty)

Personal best = 2.10m(Person ability)

Personal best = 1.72m(Person ability)

Personal best = 1.50(Person ability)

Personal best = 1.80m

(Person ability)What’s the probability he will clear the bar?50:50?Anxiety? Preparation?

Etc.

Wright map

ICC and IIC

Polytomous IRT (e.g. partial credit model)

Fit statistics to Rasch model (chi-square)

0.0

0.2

0.4

0.6

0.8

1.0

-3 -2 -1 0 1 2 3

Pro

po

rtio

n C

orr

ect

Ability

Item 1 Item 2 Item 3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

-3 -2 -1 0 1 2 3

In

form

ati

on

Ability

Item 1 Item 2 Item 3 Total

Item with more than 1 mark, e.g. 4 marks

Monotonic increase from 0 to 1 to 2 to 3 to 4 marks