Embed Size (px)
Citation preview
Proposed Cognitive Test for the Main Study
National Research Coordinators Meeting Windsor, June 2008
NRCMeetingWindsor
June 2008
• Feedback from group discussions, comments, recommendations
Contents
≅ 50min
• Group discussion of specific recommendations ≅ 60min
≅ 70 min
• First impressions
• Summary of recommendations
• Possible outcomes • Item deletions, minor changes.
NRCMeetingWindsor
June 2008
First impressions• Items worked well, can comprise single
scale (NRC 2, p10)• CIVED trend set can be scaled with and
used for single (NRC 2 pp16,17)• Test time per item seemed appropriate
according to ‘not reached’ – still to be investigated further
• No pattern of gender DIF by item type or substance (NRC 2, p12)
• Some patterns of ‘regional’ item-by-country interaction (NRC 2, p15)
NRCMeetingWindsor
June 2008
First impressions (cont.)
• Open-ended response items showed higher ‘non-attempts’ and slightly more variability (item-by-country interaction and opportunity for inter-rater reliability issues) (NRC 2, p17)
• Targeting good (NRC 2, p9)
NRCMeetingWindsor
June 2008
Open-ended response item scoring guides
• Revision:– Structural changes informed by
suggestions from countries*– Addition of “real” student responses (thank
you to Julie Nelson at NFER)– Addition of relevant scoring codes on the
advice of countries*
* Feedback from NRCs through questions, voluntary reports and field-trial operations survey.
– Finalisation of scoring guides at the marker training seminar in Hamburg
NRCMeetingWindsor
June 2008
Summary of Item Recommendations
Recommendation MCQ
Include unchanged 65 + 6*
Include only with change 11
Delete 3 + 1*
Include and consider possible change 2
Possible deletion or inclusion with change 0
Possible delete 2 + 5*
Total 90
OE
3
1
1
2
1
0
8
Total
74
12
5
4
1
7
98 *T/F
NRCMeetingWindsor
June 2008
Possible Outcomes• Shorten test to 45 minutes• Increase number of clusters to 7• Delete two open-ended response
items and approximately 15 MCQ to maintain test-taking rate.
• Some updates on meeting documents resulting from further analyses and PAC meeting.– True/False items (NRC 5, pp 25-26)– PAC recommendations/suggestions
(tabled document)
NRCMeetingWindsor
June 2008
Latest News
• Some updates on meeting documents resulting from further analyses and PAC meeting.– True/False items (NRC 5, pp 25-26)– PAC recommendations/suggestions
(tabled document)
NRCMeetingWindsor
June 2008
Part 2Group discussions of items
NRCMeetingWindsor
June 2008
Context of Group Discussions
• Assessment framework developed in consultation with NRCs and other experts before the field-trial.
• All countries invited to submit items before field-trial
• All items made available for online feedback by NRCs before the field-trial
• All items revised on the basis of the online feedback for the field-trial and approved for inclusion at the NRC meeting in Rome.
NRCMeetingWindsor
June 2008
Context for Group Discussions
• Field-trial empirical data collected from 31 countries and approx. 19, 000 students
• Recommendations for changes to instrument are based on empirical (and some qualitative) data from the field trial.
NRCMeetingWindsor
June 2008
Context for Group Discussions
• The only changes that can be made to the cognitive instrument now are:
1. Deletion of items2. Minor changes to items only if we are
very confident that a change will fix a problem (and not introduce a new problem)
3. Amendments to the scoring guides
NRCMeetingWindsor
June 2008
Context for Group Discussions
• All items with “good” data are to be included in the MS instrument
• Groups to consider each of the ISC recommendations and make one of three recommendations for each:
1. Accept definitive ISC recommendation
2. Reject definitive ISC recommendation (delete item)
3. Offer alternative “solution” to ISC recommendation
NRCMeetingWindsor
June 2008
Session 2 – Until lunch
1. 30 minutes – finish group discussion
2. 60 minutes – BRIEF presentation from each group
NOTE: Please submit all your comments and feedback to us (me) and we will include them in our deliberations regarding the final instrument.
NRCMeetingWindsor
June 2008
Part 3 Feedback from group
discussions, comments, recommendations
NRCMeetingWindsor
June 2008
Booklet rotation
Clust.Booklet
# Pos 1 Pos 2 Pos 3
A 1 A B D
B 2 B C E
C 3 C D F
D 4 D E G
E 5 E F A
F 6 F G B
G 7 G A C
NRCMeetingWindsor
June 2008
Not reached for Latin American Countries
NRCMeetingWindsor
June 2008
Recommended delete
Possible delete
3 | | | | | | |48.2 |57.2 |23.2 | 2 | X|97.2 XX| XX| XX|67 XX|57.1 68.2 XXX|83.2 XXXX|78 XXXX|31 45 1 XXXX|76 XXXXXX|17 42 80 XXXXX|12 41 48.1 72 XXXXXX|23.1 59 73 77 98.2 XXXXXXX|61 75 91 XXXXXXXX|8 79 98.1 XXXXXXXX|11 37 60 85 89 90 94 XXXXXXXXX|14 29 47 96.2 XXXXXXXX|6 53 65 70 84 92 XXXXXX|22 27 63 81 0 XXXXXXX|44 64 68.1 83.1 88 XXXXXXX|2 4 7 9 13 32 50 71 93 97.1 XXXXXXXX|15 55 62 XXXXXXXXXX|5 20 25 30 58 XXXXXXXX|16 24 26 XXXXXXX|18 43 46 49 51 52 54 82 86 XXXX|1 21 28 XXXXX|3 19 34 35 56 66 96.1 XXXXX|40 74 -1 XXX| XXX|10 69 XX|33 36 95 XX| X| X| X|38 87 X|39 | | -2 |
NRCMeetingWindsor
June 2008
Recommended delete
Possible delete
3 | | | | | | |48.2 | |23.2 | 2 | X|97.2 XX| XX| XX| XX| 68.2 XXX|83.2 XXXX| XXXX| 45 1 XXXX|76 XXXXXX|17 42 80 XXXXX|12 41 48.1 72 XXXXXX|23.1 59 73 77 98.2 XXXXXXX|61 75 91 XXXXXXXX|8 79 98.1 XXXXXXXX|11 37 60 85 89 90 94 XXXXXXXXX|14 29 47 96.2 XXXXXXXX|6 53 65 70 84 92 XXXXXX|22 27 63 81 0 XXXXXXX|44 64 68.1 83.1 88 XXXXXXX|2 4 7 9 13 32 50 71 93 97.1 XXXXXXXX|15 55 62 XXXXXXXXXX|5 20 25 30 58 XXXXXXXX|16 24 26 XXXXXXX|18 43 46 49 51 52 54 82 86 XXXX|1 21 28 XXXXX|3 19 34 35 56 66 96.1 XXXXX| 74 -1 XXX| XXX|10 69 XX|33 36 95 XX| X| X| X|38 87 X|39 | | -2 |
NRCMeetingWindsor
June 2008
Recommended delete
Possible delete
3 | | | | | | | | | | 2 | X| XX| XX| XX| XX| XXX| XXXX| XXXX| 45 1 XXXX|76 XXXXXX|17 42 80 XXXXX|12 41 72 XXXXXX| 59 73 77 XXXXXXX|61 75 91 XXXXXXXX|8 79 XXXXXXXX|11 37 60 85 89 90 94 XXXXXXXXX|14 29 47 XXXXXXXX|6 53 65 70 84 92 XXXXXX|22 27 63 81 0 XXXXXXX|44 64 88 XXXXXXX|2 4 7 9 13 32 50 71 93 XXXXXXXX|15 55 62 XXXXXXXXXX|5 20 25 30 58 XXXXXXXX|16 24 26 XXXXXXX|18 43 46 49 51 52 54 82 86 XXXX|1 21 28 XXXXX|3 19 34 35 56 66 XXXXX| 74 -1 XXX| XXX|10 69 XX|33 36 95 XX| X| X| X|38 87 X|39 | | -2 |
NRCMeetingWindsor
June 2008
True/False Items
Composite model 1
Yuck!
NRCMeetingWindsor
June 2008
True/False Items
Composite model 2
Also yuck!
NRCMeetingWindsor
June 2008
True/False Items
36: laws can be made by an elected <parliament/legislature>.
Disc: 0.18
NRCMeetingWindsor
June 2008
True/False Items37: judges make legal decisions without being influenced by the government.
Disc: 0.16
NRCMeetingWindsor
June 2008
True/False Items38: the religion of the majority is the only legal religion.
Disc: 0.36
NRCMeetingWindsor
June 2008
True/False Items38: the religion of the majority is the only legal religion.
NRCMeetingWindsor
June 2008
True/False Items
39: people have the right to criticise the government.
Disc: 0.31
NRCMeetingWindsor
June 2008
True/False Items
39: people have the right to criticise the government.
NRCMeetingWindsor
June 2008
True/False Items40: only adults who pay income tax are allowed to vote.
Disc: 0.15
NRCMeetingWindsor
June 2008
True/False Items41: all adult voters have the right to vote on new laws in the national <parliament/ legislature>.
Disc: 0.18
