Using the CEFR in Catalonia
Neus [email protected]
The EOI system
●State funded language schools (+16)
●Two levels defined (aimed at B1 and B2)
● Published curricula
●13 different languages
●40,000 students
●16,000 certificate exams every year
●Standardised certificates since 1995
Issues to be solved in 2002-1
●Revise certificate examinations.
● Is examination difficulty equivalent across time?
● Is the lower certificate consistently easier than the higher certificate?
●Are the different certificates in the different languages comparable?
Issues to be solved in 2002-2
●Develop level specifications related to the CEFR.
●Revise existing curricula in relation to the CEFR.
●Link certificates to CEFR levels.
● Project design
● Empirical scale development
● Item banking (English)
● Manual procedures for linkage, (Specification Standardisation, Empirical validation)
● Defining and exemplifying A2
● Developing curriculum objectives for A2, B1 and B2
● Developing test specifications for A2, B1 and B2
Project Overview 2003-2007
Scale Development
Item BankingLinking tests
Changing curricula
CEFR
Challenges
●Where to start?●Involve teachers (and item writers).●Improve existing practice.●Bring in the ELP onto the project.●How? Need to count on experts.●Limited resources.
Scale development (all languages)
CEFR Methodology
Step 1: selecting level descriptors.
Step 2: translation into Catalan.
Step 3: mapping descriptors onto levels.
Step 4: developing and validating new scales.
Descriptors Teachers* Languages
● Reading 40 103 10● Listening 46 99 10● Speaking 80 92 12 ● Writing 53 89 12
● Grammar 34 81 12 ● Vocabulary 61 73 12
* Arabic, Basque, Catalan, Dutch, English (>40), French (>20), German (>15), Italian, Japanese, Portuguese, Russian, Spanish
Teachers involved
Lessons learnt from scale development
●Continued training/familiarisation is necessary.
●Appearances need to be checked empirically.
●Exact correspondence may not be possible, but is it desirable?.
●Linkage does not mean equivalence.
Item Banking (English)
MethodologyStep 1
● Booklet development
Booklet design for anchoring items (2003)
S1 S2 S3 S4 S5 L1 L2 L3 A1 A2 A3
V1 X X X
V2 X X X
V3 X X X
V4 X X X
V5 X X X
V6 X X X X
V7 X X X X
V8 X X X X
MethodologyStep 2● Data collection (260-784 students per item)● Analyses: CTT and IRT (Total surviving items :301)
Step 3● Standard setting procedures: - test centered - examinee centered - annual average pass rate● Setting (provisional) cut off scores at Elemental (B1) and
Aptitud (B2)
TCC of the eight versions(2003)
-3 -2 -1 0 1 2 3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Theta
p1 p2 p3 p4 p5 p6 p7 p8
Using the Manual for linking exams (English, French, German)
Challenges in Specification
●How to map the examination? Different versions of the test? Specifications?
●How to tackle differences of coverage (subskills)?
●Who does what?
●Who checks it?
Challenges in Standardisation
●Reference “r” materials not ready until 2005.
●Differences across skills, across languages.
●Assessing task vs. item levels.
● No “linguistic competence” reference materials.
The proposal from Dutch CEFR project
●Training.●Describing texts and items according to
set parameters (reading and listening).●Estimating their CEFR levels.●Pretesting the items thus labelled.●Calibrating the items.●Standard-setting on the scale coming
from the calibration.●Assigning a psychometric level to the
items. ●Assigning a definitive level to the items.
Using the Council of Europe Item CD (German)
The German booklet
Reading Listening
StudentsB1 B2
294 307
ItemsA2 C1
40 40
JudgesNative-non native
12 10
Institutions Dialang, Goethe, WBT, TestDAF, EOI
Dialang, Goethe, WBT, EOI
Inter-judge Consistency in assessing items
R15L22
Judges: Consistency with the Pre-Estimation
1 2 3 4 5 6 7 8 9 10 11 12 13
0,0
0,2
0,4
0,6
0,8
1,0
Mean
Reading: = 0.98
Pre-est.
r tot
Judges1 2 3 4 5 6 7 8 9 10 11 12 13
0,0
0,2
0,4
0,6
0,8
1,0
Mean
Listening: = 0.94
Pre-est.
r tot
Judges
Consistency pre-estimation with Empirical Results
Developing CEFR based curricula
Developing CEFR-based curricula
●Focusing on what students can do.●Drafting objectives. ●Defining content.●Defining assessment criteria. +●Methodological guidelines.
Conclusions so far
The CEFR IS a Bible ( but only in the widest sense of the word)
The Manual is no book of spells
There is no ONE holy Grail
●We learnt much more about our exams.
●It has been a competence building process.
●We have become less dogmatic.
●We know there is further work to do and room for improvement.
●Combining highly technical work with enthusiasm is crucial.
How do I know if my B1 is your B1?
This is my B1. What’s your B1 like?