The Poisson-Gamma model for speed tests Norman Verhelst Frans Kamphuis National Institute for...

The Poisson-Gamma model for speed tests

Norman VerhelstFrans Kamphuis

National Institute for Educational Measurement Arnhem, The Netherlands

The student monitoring system

• Measurement of individual development – Common scale

• Estimation of distribution (norms) – Twice per grade (M3, E3,…,M8)

• Several subjects– Arithmetic– Reading comprehension– Technical reading

Two types of speed tests

• Basic observation is the time to complete a task– AVI cards

• Basic observation is the number of completed subtasks within the time limit– Tempotests (TT)– Three Minute Test (TMT)

Example tempotest (E4)

• Op de politieschool spelen ze ook rook koor een soort toneel

• Het lijkt wel wat op ‘politie en boefje spelen stelpen slepen’.

• Net zoals op de basisschool.

• Wat poe doe boe je bij een gevecht?

• Je pistool trekken?

• Nee, dat mag zomen zomaar zomer niet.

Example TMT

• Easy version– as– fee– oom– uur– zee– oor– …– poot (=150)

• Hard version– banden– geluid– tante– beker– kuiken– koffer– …– brandweerwagen

(=150)

Models

• Measurement model: Poisson– What is the relation between the (latent)

ability and the test performance?

• Structural model: Gamma– The distribution of the latent ability in one or

more populations? (M3, E3, M4,…,M8)

Measurement model: Poisson (1)

: observation (number read/number correct)

: student index

: task index

vix

v

i

( ; ) , ( 0,1,2,3, )!

vix

vi vivi

P x e xx

Measurement model: Poisson (2)

( ; ) , ( 0,1, 2,3, )!

vix

vi vivi

P x e xx

vi i v i

: time limit (in minutes)

: easiness of task (dimensionless)

: ability (#subtasks/minute)

i

i

v

i

Parameter estimation:incomplete design (JML)

1

statistics: en k

v vi vi i vi vii v

s d x t d x

1

normalisation: 1k

ii

vv

vi i ii

s

d

ii

i vi vv

t

d

Person parametersˆ ˆv vi i i

i

d

ˆˆv

v

v

s

ˆ |E

ˆˆ( )

ˆ ˆvv v

vv v v

sSE

is the corrected reading time (weights: )i

Design TMT

• 3 difficulty levels (1, 2, 3)

• For each level: three parallell versions (a, b, c)

• Each student participates twice: medio and end of same grade

• At each administration: 3 cards of levels 1, 2 and 3 (in that sequence)

• M3: only cards 1 and 2

voor de groepen 4-7medio eind

1 2 3 1 2 31 a a a b b b2 a a b b b c3 a a c b b a4 a b a b c b5 a b b b c c6 a b c b c a7 a c a b a b8 a c b b a c9 a c c b a a

10 b a a c b b11 b a b c b c12 b a c c b a13 b b a c c b14 b b b c c c15 b b c c c a16 b c a c a b17 b c b c a c18 b c c c a a19 c a a a b b20 c a b a b c21 c a c a b a22 c b a a c b23 c b b a c c24 c b c a c a25 c c a a a b26 c c b a a c27 c c c a a a

Two step procedure

• Estimate the task parameters σi

– JML = CML

• Estimate latent distribution while fixing the task parameters at their CML -estimate

Advantage

1 2 1 2

1 2 1 2

If and indep. Poisson with parameters en ,

then is Poisson distributed with parameter

X X

X X

[ ] ( )v vi v i i vi i

s s P P

Structural model:distribution of reading speed (θ)

1 ( ; , ) exp( )( )

g

( )E

2

( )Var

Marginal distribution of the sum score s

0

1

0

( | )

( )

( ) (

!

)

( )

s

P s

es

f s g d

e d

Negative Binomial(Gamma-Poisson)

( )( )

! ( ) ( )

s

s

sf s

s

p

1 p

( )( ) (1 )

! ( )ss

f s p ps

Negative binomial

1

0

( 1) ( )

( ) ( )( )

( ) ( )

s

j

sj

1

0( )

( ) (1 )!

s

j sj

f s p ps

EAP

| Gamma( , )s s

( | )s

E s

( | )s

SD s

Reliability

'SS p

Validation (tempo test)M4

0

5

10

15

20

25

25 50 75 100 125 150 175

Validation (tempo test)

0.00

0.25

0.50

0.75

1.00

25 50 75 100 125 150 175gobserveerde scores

exp(M4)

obs(M4)

exp(E4)

obs(E4)

Validation (TMT)

M3

0

5

10

15

20

25

30

0 50 100 150 200

Latent class model• Population consists of two latent classes

of size π and 1 - π respectively • The latent variable is gamma distributed in

each class• Parameters

– π– α1 en β1

– α2 en β2

• EM-algorithm

M3 (pi = 0.54)

0 20 40 60 80 100theta (words per minute)

class 1

class 2

mixture

Validation (TMT)

M3

0

5

10

15

20

25

30

0 50 100 150 200

Validation (TMT)

0.00

0.25

0.50

0.75

1.00

0 50 100 150 200 250aantal woorden gelezen

exp(M3)

obs(M3)

exp(E3)

obs(E3)

Norms (TMT)

0.00

0.25

0.50

0.75

1.00

0 20 40 60 80 100 120theta (= woorden per minuut)

M3 E3 M4 E4 M5 E5 M6 E6 M7 E7 M8

Thank you

Example: student vTask i dvi

1 0 8 0.93 -

2 1 8 1.11 8.88

3 0 6 0.85 -

4 1 6 1.05 6.30

5 0 5 1.09 -

δv : 15.18

i

ivi id i

122122 8.04 (subtasks/minute on a standard task)

15.18vvs

122( ) 0.73

15.18vSE

Problems

• SE(π) large

• Local maxima?

• Thick right tail of observations

• >2 classes?– Initial estimates

• Homogeneity of test material

• Local independence

Simulation E3

0

0.2

0.4

0.6

0.8

1

10 15 20 25 30 35 40

average class 1

siz

e c

las

s 1

real pi = 0.51; estimated pi = 0.93

0

200

400

600

800

1000

0 50 100 150 200 250score

cu

mu

lati

ve

fre

qu

en

cy Obs.

Exp.

Class 1 Class 2 Overall

Mean 28.15 44.07 35.99

SD 2.71 3.22 0.43

Averages (1000 replications)

Standard deviations (1000 rep.)

Class 1 Class 2 Overall

Mean 13.31 17.44 17.66

SD 2.21 1.68 0.47