View
214
Download
0
Category
Tags:
Preview:
Citation preview
Raili HildénRaili Hildén University of Helsinki University of Helsinki
Relating the Finnish School Relating the Finnish School Scale to the CEFRScale to the CEFR
Starting pointStarting point
• ““The construction of a comprehensive, The construction of a comprehensive, transparent and coherent framework transparent and coherent framework for language learning and teaching for language learning and teaching does not imply the imposition of one does not imply the imposition of one single uniform system. On the contrary, single uniform system. On the contrary, the framework should be open and the framework should be open and flexible, so that it can be applied, with flexible, so that it can be applied, with such adaptations as prove necessary, such adaptations as prove necessary, to particular situations.” (CEFR, 2001, to particular situations.” (CEFR, 2001, p. 7)p. 7)
How was the Finnish school scale (FSS) designed?
• Decision to adapt/adopt the CEFR levels was made by the language experts invited to be members of a national curriculum development working group; approved by the supervising board
• Sources consulted: • CEFR – scales for multiple communicative activities• Canadian Benchmarks • Teacher judgement • Working group members commented on the drafts
How was the school scale designed?
• Several versions were produced based on internal feedback
• First round of empirical check for inter- rater consistency (by the team members)
• Re-formulation of descriptors with low agreement/consistency
• Second round of empirical validation
• Agreeing on the current formulations
Proficiency levels (and their labels) in Finnish language curricula
http://www.oph.fiA1 Basic communication in the most familiar situations
A1.1 First stage of elementary proficiency
A1.2 Developing elementary proficiency
A1.3 Functional elementary proficiency
A2 Communication in basic social situations and simple description
A2.1 First stage of basic proficiency
A2.2 Developing basic proficiency
B1 Communication in everyday life
B1.1 Functional basic proficiency
B1.2 Fluent basic proficiency
B2 Coping with regular relationships with native speakers
B2.1 First stage of independent proficiency
B2.2 Functional independent proficiency
C1-C2 Proficient language use in demanding contexts
C1.1 First stage of skilled proficiency
Categories included in the Finnish language curricula
Listening comprehension• Themes, text and tasks
(1)• Conditions and
constraints (2)
Speaking • Themes, texts and tasks
(monologue and interaction)
• Fluency
• Pronunciation
• Linguistic range
• Linguistic control
Reading comprehension• Themes, texts and
tasks (2)• Conditions and
constraints (1)
Writing • Themes, texts and
tasks (2)• Linguistic range
• Linguistic control
Research Questions
RQ1. What is the level of agreement between judges on the CEFR level of the FSS descriptors?
RQ2. How are the Finnish syllabus descriptors related to CEFR levels?
Design
• FSS level descriptors were split up into 184 statements
• Listening 38, Speaking 66, Reading 31 Writing 49 statements
• The statements were coded and grouped in terms of communicative activities
• Criterion scales used in the rating of FSS descriptors were selected from among relevant CEF scales
Design
• A sample of 40 Finnish language teaching experts were contacted by an e-mail questionnaire
• 20 experts returned the questionnaire• A randomised selection of statements
referring to each of the four communicative activities was e-mailed to the raters (in Finnish translation by Huttunen & Jaakkola 2003)
Conducting the research
Listening comprehensiona (38 descriptors)
Raters
Sent to:Returned
by:
1. Themes and textsb (13) 40 20
2. Tasks and activities (14) 16 8
3. Conditions and constraints (11) 15 10
Notes:a/ CEFR scale used for the rating task- Overall listening comprehensionb/ Anchor descriptors, rated by all raters
Speaking (66 descriptors)
CEFR scale Raters
Sent to: Returned by:
1. Texts, themes and tasks (21)
Overall spoken interaction
40 20
2. Fluency (10) Spoken fluency 14 6
3. Pronun-ciation (10)
Phonological control
15 8
4.Linguistic range (14)
Vocabulary range 15 9
5. Linguistic control (11)
Grammatical accuracy
15 10
Conducting the researchReading comprehensiona (31 descriptors)
Raters
Sent to: Returned by:
1. Themes and textsb (10) 40 20
2. Tasks (11) a 14 6
3. Conditions and constraints (10)14 7
4. Tasks and activities c 12 7
Notes:a/ CEFR scale - Overall reading comprehensionb/ Anchor descriptors, rated by all ratersc/ CEFR scale Reading for information and argument
Writing (49 descriptors)
CEF scale Raters
Sent to: Returned by:
1. Texts and themes (13)
Creative writing 40 20
2. Tasks and activities (12)
Overall written interaction
15 10
3. Linguistic range (11)
General linguistic range
15 7
4. Linguistic control (13)
Grammatical accuracy
15 7
5. Texts and themes (12)
Overall written production
15 6
FSS level codes CEFR level codes
A11 - 1
A12 - 2
A13 - 3
A21 - 4
A22 - 5
B11 - 6
B12 - 7
B21 - 8
B22 - 9
C11 -10
A1 - 1
A2 - 2
A2+ - 3
B1 - 4
B1 + - 5
B2 - 6
B2+ - 7
C1 - 8
C2 - 9
RQ1.The range of raters’ agreement
Frequency Percent Valid Percent Cumulative
Percent 0 10 5,4 5,4 5,4 1 40 21,7 21,7 27,2 2 42 22,8 22,8 50,0 3 39 21,2 21,2 71,2 4 26 14,1 14,1 85,3 5 21 11,4 11,4 96,7 6 3 1,6 1,6 98,4 7 2 1,1 1,1 99,5 8 1 ,5 ,5 100,0
Valid
Total 184 100,0 100,0
Range
Note: Ranges of 7 and 8 were checked after the presentationand detected to be due to clerical errors. Thus the ”true” range isFrom 0 to 6.
variance=2,37 FINSS=B21 CEF=6 CEF reference B2
R12
1 5,0 5,0 5,0
7 35,0 35,0 40,0
5 25,0 25,0 65,0
7 35,0 35,0 100,0
20 100,0 100,0
4,00
6,00
8,00
9,00
Total
ValidFrequency Percent Valid Percent
CumulativePercent
RQ 1.Agreement with the CEF levelsExample of a descriptor with a low level of agreement between raters.“Can identify the writer’s bias and the purpose of the text and locate and integrate several specific pieces of information in a longer text. Can quickly identify the content and relevance of new items deciding whether closer study is worthwhile”
Level
variance=0 FINSS=A11 CEF=1 CEF reference A1
RQ 1.Agreement with the CEF levels Example of a descriptor with a high level of agreement between raters. “Can write the alphabet of the language and all numbers and numerals. Can write down basic personal identification information and write a small number of familiar words and simple phrases.”
W42
11 55,0 100,0 100,0
9 45,0
20 100,0
1,00Valid
SystemMissing
Total
Frequency Percent Valid PercentCumulative
Percent
RQ2. Agreement between syllabus level descriptors (syllcode) and the
original CEFR levels (levcode)
levcode
A1 A2 B1 B2 C1 C2 Total
A1 28 18 1 0 0 0 47 A2 2 25 8 0 0 0 35 B1 0 2 26 6 0 0 34 B2 0 0 4 28 12 1 45
syllcode
C1 0 0 0 11 11 1 23 Total 30 45 39 45 23 2 184
Note: A quite good level of agreement was oberved (65%). There is, however, some tendency for an overestimation: 19 descriptors (10%) were assigned to a lower level while 46 (25%) were assigned to higher level.
RQ1. Range distribution per skill
WSRL
skill
8
7
6
5
4
3
2
1
0
range
The average range is 2.66 for the whole pool. The range distribution per skill can be seen in the boxplot above, showing no clear differences between the four skills.
RQ2. Agreement between individual rating and original (initial) levels
(Syllabus - syll & CEF - level)
Plans for further exploration
• Calibrating the FSS descriptors• Exploring the link to the Canadian
Benchmarks in more detail• Re-formulating or removing problematic
descriptors• Empirical validation and exemplication of
the FSS scales through benchmarks for comprehension tasks and for oral and written performance samples
Summary
• The correspondence between the new Finnish school scale (FSS) and the CEFR scales were studied.
• Research question 1: What is the level of agreement between judges on the CEFR level of the FSS descriptors?
• Research question 2: How are the Finnish syllabus descriptors related to CEFR levels?
• 20 experienced raters judged FSS descriptors using relevant CEFR scales
• A good agreement was reached: 65% of the FSS descriptors were assigned to the original CEFR levels. For the rest of the cdescriptorts, some tendency of overestimation was observed.
• Inter-rater agreement was also quite good.
Recommended