Upload
kelvinyong
View
239
Download
0
Embed Size (px)
Citation preview
7/21/2019 Timss Model Kbat
1/61
TIMSS International ResultsTIMSS Released Items
TIMSS Technical Report
http://timssandpirls.bc.edu/timss2011/international-results-mathematics.html
Personal Communication with IEA
S.Kanageswari Suppiah Shanmugam, PhD.
7/21/2019 Timss Model Kbat
2/61
TIMSS
Trends in International Mathematics and Science
Study International assessment on Maths/Science for
grade 4/Grade 8
Conducted by International Association forEvaluation of Educational Achievement (IEA)
Every four years since 1995 (TIMSS 2011- 5th cycle)
Development and Administration Maths/Sc Test Booklets
Student/Teacher/School/Curriculum Questionnaires
2
7/21/2019 Timss Model Kbat
3/61
Maths Questionnaires
Student Questionnaire(S)
Home Experience School Experience
Teacher Questionnaire (T) Education
Professional Development
Teaching Experience
School Questionnaire(P)Availability of Resources
Types of Programmes
Learning Environment
Curriculum Questionnaire(C) Organisation of Maths/Sc Curriculum
Content of Maths/Sc Curriculum
3
7/21/2019 Timss Model Kbat
4/61
Student Testing Time
Student Achievement Booklet Part 1 45 minutes
Break
Student Achievement Booklet Part 2 45 minutes
Break
Student Questionnaire 30 minutes
4
7/21/2019 Timss Model Kbat
5/61
Student Achievement Booklet28 item blocks 14 Maths
14 Sc Assigned to 14 Booklets
Each booklet has 4 blocks 2 maths 2 Sc
Each block has 12- 18 items and in two booklets(Linking) MCQ Structured Response Questions
- Provide explanation- Support an answer with reasons- Numerical evidence- Draw diagram- Display data
One block- score points of 18 (on average)Maths/Sc blocks alternately begin (balance position
effect) 5
7/21/2019 Timss Model Kbat
6/61
TIMSS Mathematics AssessmentFramework
6
7/21/2019 Timss Model Kbat
7/61
TIMSS 2011 Mathematics Framework
is organized around two dimensions
a content dimension
specifying the domains or subject matter to be assessedwithin mathematics (number, algebra, geometry, and
data and chance)Has several topic areas
Each topic area is presented as a list of objectives
a cognitive dimension
specifying the domains or thinking processes to beassessed (knowing, applying, and reasoning)
The cognitive domains describe the sets of behaviorsexpected of students as they engage with the
mathematics content.
7/21/2019 Timss Model Kbat
8/61
TIMSS Maths Content Domain
Number (30%) Algebra
(30%)Whole NumFractions/DecimalsIntegersRatio/Proportion/Percent
PatternsAlgebraic ExpressionsEquation/Formula and Function
Geometry(20%)
Data & Chance(20%)
G ShapesG MeasurementLocation & Movement
Data organisation &representationData InterpretationChance
8
7/21/2019 Timss Model Kbat
9/61
Using mathematics Depends on mathematical knowledge Familiarity with mathematics concept Facts - factual knowledge that provides the basic
language of mathematics, and the essential
mathematical facts and properties that form thefoundation for mathematical thought. Procedures- entails recall of sets of actions and
how to carry them out & computationalprocedures and tools
Knowledge of concepts - make connectionsbetween elements of knowledge, judge the validityof mathematical
statements and methods, and create mathematicalrepresentations
Cognitive Domain
Knowing(35%)
7/21/2019 Timss Model Kbat
10/61
application of mathematical tools in arange of contexts.
The facts, concepts, and procedures oftenare very familiar to the student, with theproblems being routine ones.
apply mathematical knowledge of facts,
skills, and procedures or understanding ofmathematical concepts to createrepresentations
Problem solving is central but the problem
settings are more routine in theimplemented curriculum.
have been standard in classroom exercises
textbook problems
Applying
(40%)
Cognitive Domain
set in real-life situations
purelymathematicalquestionsemphasismore familiarand routine
tasks
7/21/2019 Timss Model Kbat
11/61
for logical, systematic thinking.
includes intuitive and inductive reasoning
based on patterns and regularities that can beused to arrive at solutions to non-routine
problems. Non-routine problems - problems that are
very likely to be unfamiliar to students.
They make cognitive demands over andabove those needed for solution of routine
problems, even when the knowledge andskills required for their solution have beenlearned.
Cognitive DomainReasoning(25%)
noveltyof the contextthe complexityof thesituation,any solution to theproblem must involveseveral steps, drawing on knowledgeand understandingfrom different areas ofmathematics
involve transfer ofknowledge and skills tonew situationsInteractions amongreasoning skills
7/21/2019 Timss Model Kbat
12/61
Cognitive Domain
Knowing
(35%)
Applying(40%)
Reasoning(25%)
12
7/21/2019 Timss Model Kbat
13/61
13
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN KNOWING
7/21/2019 Timss Model Kbat
14/61
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN KNOWING
14
7/21/2019 Timss Model Kbat
15/61
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING
15
7/21/2019 Timss Model Kbat
16/61
16
TOPIC AREA FRACTIONS AND DECIMALS
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING
7/21/2019 Timss Model Kbat
17/61
TOPIC AREA RATIO, PROPORTION AND PERCENT
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN APPLYING
17
7/21/2019 Timss Model Kbat
18/61
18
TOPIC AREA GEOMETRIC SHAPES
CONTENT DOMAIN GEOMETRY
COGNITIVE DOMAIN REASONING
7/21/2019 Timss Model Kbat
19/61
19
TOPIC AREA INTEGER
CONTENT DOMAIN NUMBER
COGNITIVE DOMAIN REASONING
7/21/2019 Timss Model Kbat
20/61
20
TOPIC AREA GEOMETRIC SHAPES
CONTENT DOMAIN GEOMETRY
COGNITIVE DOMAIN REASONING
7/21/2019 Timss Model Kbat
21/61
TOPIC AREA DATA
CONTENT DOMAIN DATA AND CHANCE
COGNITIVE DOMAIN REASONING
21
7/21/2019 Timss Model Kbat
22/61
INTERNATIONAL BENCHMARKADVANCED (625)
-reasonwith information, draw conclusions, make generalizations, and solve linearequations.-solve a variety of fraction, proportion, and percent problems andjustify
conclusions.-express generalizations algebraically and model situations.-solve a variety of problems involving equations, formulas, and functions.-reason with geometric figures to solve problems and with data from several sourcesor unfamiliar representations to solve multi-step problems
Content Domain: Geometry
Cognitive Domain: Reasoning
Description: Solves a wordproblem involving filling athree-dimensional shape with
rectangular solids
22
7/21/2019 Timss Model Kbat
23/61
INTERNATIONAL BENCHMARKHIGH (550)
-applyunderstanding and knowledge in complex situation-use information from several sources to solve problems involving differenttypes of numbers and operations.
-relate fractions, decimals, and percents to each other.-show basic procedural knowledge related to algebraic expressions.-use properties of lines, angles, triangles, rectangles, and rectangular prisms tosolve problems.-analyse data in a variety of graphs.
Peter, James, and Andrew each had 20 tries at throwing ballsinto a basket. Complete the missing boxes below.
Name Number of Percentage ofSuccessful Shots Successful
Shots
Peter 10 out of 20 50 %
James 15 out of 20
Andrew out of 20 80%
Content Domain:Number
Cognitive Domain:Knowing
Description: Given
the part and thewhole, can expressthe part as apercentage, andgiven the whole andthe %, can find the
part 23
7/21/2019 Timss Model Kbat
24/61
-apply basic mathematical knowledge in a variety of situations.-solve problems involving decimals, fractions, prop,%
-understand simple algebraic relationships.-relate a two-dimensional drawing to a three-dimensional object.-read, interpret, and construct graphs and tables.-recognise basic notions of likelihood.
INTERNATIONAL BENCHMARKINTERMEDIATE (475)
What does xy + 1 mean?o a Add 1 to y, then multiply by x.
o b Multiply x and y by 1.
o c Add x to y, then add 1.
od Multiply x by y, then add 1.
Content Domain: Algebra
Cognitive Domain:Knowing
Description: Knows the
meaning of a simplealgebraic expressioninvolving multiplicationand addition
24
7/21/2019 Timss Model Kbat
25/61
Content Domain:Number
Cognitive Domain:Knowing
Description: Adds atwo-place and athree-place decimal
Have some knowledge of whole numbers and decimals,
operations, and basic graphs
INTERNATIONAL BENCHMARKLOW (440)
42.65 + 5.748 =
Answer: ________
25
7/21/2019 Timss Model Kbat
26/61
Malaysian Students Performance
26
7/21/2019 Timss Model Kbat
27/61
Administration in Malaysia45 countries
Four countries in SEA(Indonesia, Malaysia, Singapore, Thailand)
180 Schools
5733 Form Two students
Test Booklets Bahasa Malaysia and English
Student Questionnaire Bahasa Malaysia
Maths Teacher Questionnaire English
School Questionnaire English
27
7/21/2019 Timss Model Kbat
28/61
Malaysia Performance Trend
Only in Grade 8 Participated since 1999 1999- 16th (39 countries) - BM 2003- 10th (48 countries) - BM 2007- 20th (46 countries) - BM/ENG 2011- 26th (45 countries) - BM/ENG
Recorded as one of the countries that declined greatly (40points or more)
519
508
474
440
400
420
440
460
480
500
520
540
1999 2003 2007 2011
28
7/21/2019 Timss Model Kbat
29/61
494
455
474
459
474
380
400
420
440
460
480
500
Number Algebra Geometry Data & Chance Overall
451
430 432 429
440
2011
Performance Trend- Content Domain
Num Geometry Data & Chance / Algebra
Num OverallGeometry / Data & Chance / Algebra Overall 29
7/21/2019 Timss Model Kbat
30/61
Performance Trend - Cognitive Domain
Low performance in Reasoning
Not much difference between Knowing/Applying
473
477
466
474
415420
425
430
435
440445
450
455
460
465
470
475
480
Knowing Applying Reasoning Overall
2007
444
439
426
440
2011
30
7/21/2019 Timss Model Kbat
31/61
Performance Trend -Content Domain by gender
Girls Overall Boys
460
440438
435
410
415
420
425
430
435
440
445
450
455
460
465
470
Number Algebra Geometry Data&Chance
Girls
441
419 425
422
Boys
451
430432
428
Overall
31
P f T d C i i D i
7/21/2019 Timss Model Kbat
32/61
Performance Trend- Cognitive Domainby gender
Girls Overall Boys
456
445
432
415420
425
430
435
440
445
450
455
460
Knowing Applying Reasoning
Girls
431 432
420
Bo s
444
439
Overall
32
7/21/2019 Timss Model Kbat
33/61
ACHIEVEMENT FOR MALAYSIA AT INTERNATIONAL BENCHMARK
At every benchmark, there is a decline
93
70
36
10
0
10
20
3040
50
60
70
80
90
100
Low Intermediate High Advanced
1999
93
66
30
6
200382
50
18
2007
65
36
13
2
2011
33
7/21/2019 Timss Model Kbat
34/61
388
455
507
562
630
0
100
200
300
400
500
600
700
5th 25th 50th 75th 95th
2003319
416
475
530
608
2007
290
373
440
507
589
0
100
200
300
400
500
600
700
5th 25th 50th 75th 95th
2011
Percentile points for Malaysia
34
7/21/2019 Timss Model Kbat
35/61
290
319
373
440
507
560
589
0
100
200
300
400
500
600
700
800
5th 10th 25th 50th 75th 90th 95th
Malaysia
453
494
559
620
672
713734
Singapore
PERCENTILE POINTS for MALAYSIA & SINGAPORE
in TIMSS 2011
35
7/21/2019 Timss Model Kbat
36/61
TIMSS Mathematics Items
36
7/21/2019 Timss Model Kbat
37/61
Question Types and Scoring Procedures
2 question formats- MCQ & constructed-response
Each MCQ is worth 1 score point.
Constructed-response questions worth 1 or 2 scorepoints, depending on the nature of the task and the
skills required to complete it. The choice of item format depends on the mathematics
or being assessed, and the format that best enablesstudents to demonstrate their proficiency.
M lti l Ch i Q ti (MCQ)
7/21/2019 Timss Model Kbat
38/61
Multiple-Choice Questions (MCQ) MCQ provide students with 4 response options.
Select one correct response
can be used to assess any of the behaviors in the cognitivedomains.
allow valid, reliable, and economical measurement of a widerange of content in a relatively short testing time.
do not allow for students explanations or supporting statements less suitable for assessing students ability to make more
complex interpretations or evaluations.
linguistic features need to be appropriate.
questions are written clearly and concisely. response options also are written to minimise the reading
load of the question.
The options that are incorrect are written to be
plausible, but not deceptive.
7/21/2019 Timss Model Kbat
39/61
Constructed-Response Questions (CRQ)
students are required to construct/give a writtenresponse, rather than select a response from a set of options
allow students to provide explanations, support an answerwith reasons or numerical evidence, draw diagrams, ordisplay data,
well-suited for assessing aspects of knowledge and skillsthat require students to explain phenomena or interpretdata based on their background knowledge and experience
real-world setting - the setting is familiar to students.
need to prepare a Scoring Rubric
i b i
7/21/2019 Timss Model Kbat
40/61
Scoring Rubric describes the essential features of appropriate and complete
responses.
focus on evidence of the type of behavior the question assesses.
describe evidence of completely correct, partially correct andcompletely incorrect responses.
student responses at each level of understanding provideimportant guidance to those who will be rating the studentsresponses.
In scoring, the focus is solely on students achievement withrespect to the topic being assessed, not on their ability to write
well. students need to communicate clearly
ypes o tems- ra e
7/21/2019 Timss Model Kbat
41/61
ypes o tems- ra e
T f It G d 8
7/21/2019 Timss Model Kbat
42/61
Types of Items- Grade 8
7/21/2019 Timss Model Kbat
43/61
Scoring Rubric
T f It G d 8
7/21/2019 Timss Model Kbat
44/61
Types of Items- Grade 8
T f It G d 8
7/21/2019 Timss Model Kbat
45/61
Types of Items- Grade 8
T pes of Items Grade 8
7/21/2019 Timss Model Kbat
46/61
Types of Items- Grade 8
7/21/2019 Timss Model Kbat
47/61
TIMSS Item Writing Process
47
Item Writing Process and Guidelines
7/21/2019 Timss Model Kbat
48/61
Item Writing Process and Guidelines
consider the timing, grade appropriateness, difficultylevel, potential sources of bias (cultural, gender, orgeographical, ).
Make sure that item validity is not affected by factors thatunnecessarily increase the difficulty of the item, such asunfamiliar or difficultvocabulary, grammar, directions, contexts, or stimulusmaterials
be sensitive to the possibility of unintentionally placingparticular groups of students at an unfair disadvantage
diagrams and graphs are drawn accurately (to scale unlessotherwise noted), and are correctly and fully labeled.
What should the student know? What should the student be able to do?
MCQ
7/21/2019 Timss Model Kbat
49/61
MCQ
Stem is the initial part of the item in which the task is defined. Options refer to the entire set of labeled response choices presented
under the stem. Key is the correct response option. Distracters are the incorrect response options. ask a direct question with only one correct answer, and provide plausible
distracters the question must be able to stand alone, and be answerable without
the response options do not include extraneous information that can confuse students avoid questions for which a wrong method yields the correct answer
(e.g., a question about a circle with a radius of 2, since computing either
area or circumference get 4) avoid writing items where students can work backwards from the
response options to find the correct answer (e.g., solving for x in anequation).(CRQ is more appropriate)
CRQ
7/21/2019 Timss Model Kbat
50/61
CRQWrite a full credit
answer in terms of the
language, knowledge, and skills that studentcould be expected topossess & determine
whether to allocate 1 or2 score points.
Develop a specificscoring guide
7/21/2019 Timss Model Kbat
51/61
Take Home information
Each of the items needs to contribute to the overallmathematics test
Some relatively easy items and some challenging items
Avoid items that almost all students or almost no
students are able to answer correctly
7/21/2019 Timss Model Kbat
52/61
Checklist- MCQ
7/21/2019 Timss Model Kbat
53/61
Checklist-CRQ
7/21/2019 Timss Model Kbat
54/61
Each item
1The Content Domain, topic area, and objective theitem measures
2. The Cognitive Domain
3. The item number (1, 2, 3, etc.)
4. The key (multiple-choice items only) or
5. The scoring guide
1
7/21/2019 Timss Model Kbat
55/61
1
2
7/21/2019 Timss Model Kbat
56/61
2
7/21/2019 Timss Model Kbat
57/61
7/21/2019 Timss Model Kbat
58/61
7/21/2019 Timss Model Kbat
59/61
7/21/2019 Timss Model Kbat
60/61
7/21/2019 Timss Model Kbat
61/61