The Poisson-Gamma model for speed tests
Norman VerhelstFrans Kamphuis
National Institute for Educational Measurement Arnhem, The Netherlands
The student monitoring system
• Measurement of individual development – Common scale
• Estimation of distribution (norms) – Twice per grade (M3, E3,…,M8)
• Several subjects– Arithmetic– Reading comprehension– Technical reading
Two types of speed tests
• Basic observation is the time to complete a task– AVI cards
• Basic observation is the number of completed subtasks within the time limit– Tempotests (TT)– Three Minute Test (TMT)
Example tempotest (E4)
• Op de politieschool spelen ze ook rook koor een soort toneel
• Het lijkt wel wat op ‘politie en boefje spelen stelpen slepen’.
• Net zoals op de basisschool.
• Wat poe doe boe je bij een gevecht?
• Je pistool trekken?
• Nee, dat mag zomen zomaar zomer niet.
Example TMT
• Easy version– as– fee– oom– uur– zee– oor– …– poot (=150)
• Hard version– banden– geluid– tante– beker– kuiken– koffer– …– brandweerwagen
(=150)
Models
• Measurement model: Poisson– What is the relation between the (latent)
ability and the test performance?
• Structural model: Gamma– The distribution of the latent ability in one or
more populations? (M3, E3, M4,…,M8)
Measurement model: Poisson (1)
: observation (number read/number correct)
: student index
: task index
vix
v
i
( ; ) , ( 0,1,2,3, )!
vix
vi vivi
P x e xx
Measurement model: Poisson (2)
( ; ) , ( 0,1, 2,3, )!
vix
vi vivi
P x e xx
vi i v i
: time limit (in minutes)
: easiness of task (dimensionless)
: ability (#subtasks/minute)
i
i
v
i
Parameter estimation:incomplete design (JML)
1
statistics: en k
v vi vi i vi vii v
s d x t d x
1
normalisation: 1k
ii
vv
vi i ii
s
d
ii
i vi vv
t
d
Person parametersˆ ˆv vi i i
i
d
ˆˆv
v
v
s
ˆ |E
ˆˆ( )
ˆ ˆvv v
vv v v
sSE
is the corrected reading time (weights: )i
Design TMT
• 3 difficulty levels (1, 2, 3)
• For each level: three parallell versions (a, b, c)
• Each student participates twice: medio and end of same grade
• At each administration: 3 cards of levels 1, 2 and 3 (in that sequence)
• M3: only cards 1 and 2
voor de groepen 4-7medio eind
1 2 3 1 2 31 a a a b b b2 a a b b b c3 a a c b b a4 a b a b c b5 a b b b c c6 a b c b c a7 a c a b a b8 a c b b a c9 a c c b a a
10 b a a c b b11 b a b c b c12 b a c c b a13 b b a c c b14 b b b c c c15 b b c c c a16 b c a c a b17 b c b c a c18 b c c c a a19 c a a a b b20 c a b a b c21 c a c a b a22 c b a a c b23 c b b a c c24 c b c a c a25 c c a a a b26 c c b a a c27 c c c a a a
Two step procedure
• Estimate the task parameters σi
– JML = CML
• Estimate latent distribution while fixing the task parameters at their CML -estimate
Advantage
1 2 1 2
1 2 1 2
If and indep. Poisson with parameters en ,
then is Poisson distributed with parameter
X X
X X
[ ] ( )v vi v i i vi i
s s P P
Structural model:distribution of reading speed (θ)
1 ( ; , ) exp( )( )
g
( )E
2
( )Var
Marginal distribution of the sum score s
0
1
0
( | )
( )
( ) (
!
)
( )
s
P s
es
f s g d
e d
Negative Binomial(Gamma-Poisson)
( )( )
! ( ) ( )
s
s
sf s
s
p
1 p
( )( ) (1 )
! ( )ss
f s p ps
Negative binomial
1
0
( 1) ( )
( ) ( )( )
( ) ( )
s
j
sj
1
0( )
( ) (1 )!
s
j sj
f s p ps
EAP
| Gamma( , )s s
( | )s
E s
( | )s
SD s
Reliability
'SS p
Validation (tempo test)M4
0
5
10
15
20
25
25 50 75 100 125 150 175
Validation (tempo test)
0.00
0.25
0.50
0.75
1.00
25 50 75 100 125 150 175gobserveerde scores
exp(M4)
obs(M4)
exp(E4)
obs(E4)
Validation (TMT)
M3
0
5
10
15
20
25
30
0 50 100 150 200
Latent class model• Population consists of two latent classes
of size π and 1 - π respectively • The latent variable is gamma distributed in
each class• Parameters
– π– α1 en β1
– α2 en β2
• EM-algorithm
M3 (pi = 0.54)
0 20 40 60 80 100theta (words per minute)
class 1
class 2
mixture
Validation (TMT)
M3
0
5
10
15
20
25
30
0 50 100 150 200
Validation (TMT)
0.00
0.25
0.50
0.75
1.00
0 50 100 150 200 250aantal woorden gelezen
exp(M3)
obs(M3)
exp(E3)
obs(E3)
Norms (TMT)
0.00
0.25
0.50
0.75
1.00
0 20 40 60 80 100 120theta (= woorden per minuut)
M3 E3 M4 E4 M5 E5 M6 E6 M7 E7 M8
Thank you
Example: student vTask i dvi
1 0 8 0.93 -
2 1 8 1.11 8.88
3 0 6 0.85 -
4 1 6 1.05 6.30
5 0 5 1.09 -
δv : 15.18
i
ivi id i
122122 8.04 (subtasks/minute on a standard task)
15.18vvs
122( ) 0.73
15.18vSE
Problems
• SE(π) large
• Local maxima?
• Thick right tail of observations
• >2 classes?– Initial estimates
• Homogeneity of test material
• Local independence
Simulation E3
0
0.2
0.4
0.6
0.8
1
10 15 20 25 30 35 40
average class 1
siz
e c
las
s 1
real pi = 0.51; estimated pi = 0.93
0
200
400
600
800
1000
0 50 100 150 200 250score
cu
mu
lati
ve
fre
qu
en
cy Obs.
Exp.
Class 1 Class 2 Overall
Mean 28.15 44.07 35.99
SD 2.71 3.22 0.43
Averages (1000 replications)
Standard deviations (1000 rep.)
Class 1 Class 2 Overall
Mean 13.31 17.44 17.66
SD 2.21 1.68 0.47