1

Hands On Session

With (Real) Data

2

CELPT Testing

Language proficiency written test

Singapore based

Ngee Ann Polytechnic

Students with varied cultural backgrounds

Profile reporting by language category

Calibrated bank of some 1500 questions

3

Two is four

Each CELPT test becomes two tests

Limited attempt to tailor questions

Match difficulty of questions

And candidate ability

4

How does it work?

Each test has 220 questions

Starter test of 20 questions

Branching on 20-question test score

Easier test

More difficult test

Only a single branch point

5

In a little more detail

Questions 1 to 20 Starter Test

Questions 21 to 60 Part 1A Test

Questions 61 to 100 Part 1B Test

Questions 101 to 160 Part 2A Test

Questions 161 to 220 Part 2B Test

6

Hands On – The basic task

Have two tests

21001 and 21002

Break down into four tests

21001 and 71001

21002 and 71002

Wish to build a single item bank

7

Hands On – The basic task

Need to calibrate all questions

Into a single bank

How many questions?

2 x 220 = 440? 4 x 120 = 480?

Some common questions!

Starter and others…

8

How should the task be tackled?

Four different tests – 4 x 120 questions

Four different sets of students

Key is the common questions

Use questions to link during analysis

But how? Many ways to do this!

9

Hands on soon!

Task for the afternoon

Does it matter how to do this?

Could bank one test at a time

Adding tests one by one

Could all tests be banked together?

10

IBS – Item Banking System

Takes in data from many tests

Finds common questions

Includes existing bank if appropriate

Analyses to find optimum statistics

Banks the end results

11

Hands On – for ‘Test’ 21002

IBS analysis of test 21001

IBS analysis of test 71001 and bank

One group

Plot difficulty estimates

For common questions in tests

One set from test one from bank

12

Hands On – for ‘Test’ 21002

Also a further IBS analysis

Tests 21001 and 7001

Second group

Plot difficulty estimates

For common questions in banks

For each type of analysis

13

Task again

Test 21001 Bank 1

Test 71001 + Bank 1 Bank 2

Test 21001 + Test 71001 Bank 3

Group 1: 71001 and 21001 (Bank 1)

Group 2: Bank 2; Bank 3

14

Let’s look at FIT in 21002/71002

CE0888 – Link fit = 4.22

In 21002 - δ/σ = 1.57/0.21; fit -3.78

In 71002 - δ/σ = -1.21/0.31; fit -0.52

In 21002, item is 57/60 in Part 1A

in 71002, item is 39/60 in Part 1B

Omits high in 1A

Question is probably OK

15

Question CE0888 – 57/1A/I/ATeenagers today are driven by _____________

to do a lot of things they would otherwise not do.

A. peer group pressure

B. a peer group pressure

C. the peer group pressure

D. some peer group pressure

16

Let’s look at FIT in 21002/71002

CE7414 – Link fit = -5.27

In 21002 - δ/σ = 1.87/0.21; fit 5.22

In 71002 - δ/σ = 2.01/0.14; fit 5.07

In 21002, item is 15/60 in Starter

in 71002, item is 15/60 in Starter

Bad question; unstable difficulty

17

Question CE7514 – 15/ST/H/CThere are many reasons for our losses.

They include the following :-

__________________________________________;The raw materials we bought were not equal to those specified.

A. Violation of set procedures by staff in the assembly plant

B. The violation of set procedures in the assembly plant by staff

C. Set procedures were violated by staff in the assembly plant

D. Violating set procedures being common in the assembly plant

18

Still need to bank four tests

Test 21001 Bank 1

Test 71001 + Bank 1 Bank 2

Test 21002 + Bank 2 Bank 4

Test 71002 + Bank 3 Bank 5

21001, 71001, 21002, 71002 Bank 6

19

Hands On – for all two/four tests

Now have a bank built in steps - Bank 5

Bank has questions from all four tests

Also have a bank built in one pass

Bank 6 also has all questions?

20

Hands On – for all tests together

Plot difficulty estimates

Bank 5 vs bank 6

Four groups for plotting

CE0000 - CE2999, CE3000 - CE49000

CE5000 - CE6999, CE7000 - CE7999

Plot any questions found common

21

Hands On – for all tests together

Statistical estimation – how robust?

What of fit?

Fit within is important

Fit between is also very important

Different groups of students

Fit will be a reflection of many factors

22

More general points

Joint approach to calibration is preferable

Balances (smoothes) lumpy data

Gives a better overall idea of ‘reality’

Helps to identify real problems

Measurement model makes this possible

23

The analysis of many tests

Analysis has to be possible – connectivity

Design needs consideration

Subtests – questions are only in one subtest

Largest possible group of questions

that occurs in unique grouping of tests

Q1, Q4 and Q6 are in tests X and Y

24

Linking

With four tests maximum is:

4C1 + 4C2 + 4C3 + 4C4

Or 4 + 6 + 4 + 1 = 15

All 15 are found in this analysis

Provides test by test linking

Some subtests very small

25

Fit again

Misfitting people

Misfitting questions

All due to question/person interactions

Not independent

Examples

26

Where have we been today?

Seeking to learn more about what we do

Wishing to measure not just report

Looking at inconsistent behaviour

Trying to understand our data

Aiming to build better tests

27

An Impossibility!

How can the Rasch Model apply fully?

The formulation is so strict!

But it is a system to provide measurement

If it fails, then at least we can know about it

But hang on, the model failing?

More like the data not fitting….

28

So here is the rub…

The test constructor needs to be aware

Of what is required and what has happened

Where the data have come from

Which students were used, what questions

The model will help to interpret of data

In the end, it all depends on the user

29

In Conclusion

It has been a romp

And there is much more to say and do

Plenty of books

Plenty of analysis programs

Rasch community

30

In Conclusion

Bond, T.G and Fox, C.M

Applying the Rasch Model: Fundamental Measurement in the Human Sciences

Lawrence Erlbaum Associates

ISBN: 0-8058-4252-7