1

Use of qualitative methods in relating exams to the

Common European Framework: What can we

learn?

Spiros Papageorgiou

Lancaster University

The Third Annual Conference of EALTAKrakow, Poland, 19 – 21 May, 2006

2

EALTA Guidelines for Good Practice in Language Testing and Assessment LINKAGE TO THE COMMON EUROPEAN

FRAMEWORK What evidence is there of the quality of

the process followed to link tests and examinations to the Common European Framework?

Have the procedures recommended in the Manual and the Reference Supplement been applied appropriately?

Is there a publicly available report on the linking process?

3

Overview

Linking exams to the CEFR Use of qualitative methods Verbal protocol analysis Small group research Examples from two studies Conclusion: From theory to practice

4

Linking exams to the CEFR

A process involving judgements by a panel Familiarisation Specification Standardisation

5

The Judges’ role in linking exams to the CEFR

CEFR

Manual

Test

Linking claim

Judges

6

Using qualitative methods: Aims Judges can provide insights into:1. Decision-making2. The CEFR scales3. The linking process How important is it to gain insights in all

these? Validity of the linking claim

7

Study 1: Verbal Protocol Analysis-Lancaster University

Familiarisation task from the Manual Framework: Ericsson and Simon (1993) Design: Green (1998), Banerjee (2004) 6 informants 3 sets of descriptors (Kaftandjieva and

Takala, 2002) from Table 2, CEFR 2 task layouts

8

Issues for the design of the study Training Language Kind of verbal reports Sample size Length Coding Task Procedure

9

Terminology problems/lack of definition (1)

R5 I can understand texts that consist mainly of job-related language. (B1)

“understand is such a vague term!” (I4)

10

Terminology problems/lack of definition (2) W6 I can write short simple messages

relating to matters in areas of immediate need. (B1)

“How short is short?” (I5)

11

Descriptor units-stand alone nature (1) R2 I can read articles concerned with

contemporary problems in which the writers adopt particular stances or viewpoints. (B2)

R13 I can read reports concerned with contemporary problems in which the writers adopt particular stances or viewpoints. (B2)

“to me an article is an easier text type to read rather than a report.” (I1)

12

Descriptor units-stand alone nature (2) W11 I can describe impressions. (B1) W15 I can describe experiences. (B1)

“experiences is more than impressions. It’s got feelings and images. So, because impressions is only feelings and also the description is difficult, so that’s why I gave it B2 and not B1”. (I2)

13

Key words L10 I can understand films without too

much effort. (C1)

“without too much effort. So, it comes natural, I don’t have to struggle. So, I thought it is C1.” (I3)

“that is the sort of thing a B1 person should be able to do. The key phrase I think that helped me was without too much effort.” (I6)

14

Inconsistencies “TV programmes”: in B1 and B2 Listening

only

“this […] talks about TV news programmes when there’s already one talking about TV news programmes in B1 so it has to go somewhere else. Whether it should have gone the other side of B1 whether it is even lower I am not sure”. (I6)

15

Summary of findings-VPA Interaction with the scales Decision making Problems with wording of descriptors Validity of the linking claim during

Familiarisation

16

Study 2: Small Group Research-London Specification of the Trinity College London

GESE and ISE suites 10 participants Framework: Small group research

(Davidson and Lynch, 2002) Specification Forms of the Manual Recordings of group work 3.5 hours of discussion fully transcribed Atlas.ti Aim: validity of Specification claim

17

Research design issues Choice of participants How to group them Roles: what the individuals do and how

they communicate within the group Cohesiveness: fitting together of group

members How can these affect the validity of the

linking claim? What other factors might exist?

18

Fitness of descriptors Descriptors do not always match the test

content532 Tim can make him/herself understood in very

short utterances, even though pauses, false starts and reformulation are very evident. can expand learned phrases through simple recombinations of their elements

533 Lora no 533 Tim we are not testing that 534 Lora so it is up to evident 535 Tim that’s right

19

Prior knowledge/Bias of insiders Judges likely to see the test through

claims already made Decision making: confirmatory rather

than exploratory

046 Lora look at the B1 under the global scale on page 4

047 Tim yes 047 Lora yes B1 048 Tim it pretty well fits [.]

20

Predicting behaviour of learners Major factor in decision making:

predicting what test takers can do

080 Lora I think they can explain a viewpoint but not necessarily the advantages and disadvantages.

081 Tim explain a viewpoint but not give advantages and disadvantages. ok?

21

Test specifications Test specifications: very influential

644 Lora shopping 645 Tim shopping facilities food

stuff clothes and fashion 646 Lora um no 647 Tim but then we do expect them

to be able to identify clothes at Grade 1 so we’ve got to say yes

22

CEFR and Manual: problems during Specification Terminology problems/lack of definition Tasks and communicative tasks Evidence vs. inferences Real life vs. exam environment Young learners Not everything is scaled Purposes and functions

23

Summary of findings-SGR The Specification process: not without

problems Group dynamics Validity of the linking claim during

Specification Transparency

24

Conclusion: From theory to practice Linking claim and decision making:

Validity, consequences, ethics Qualitative & quantitative analysis of

judgements: complementary Learn more about the use of the

Framework: linking exams and transparency

Validation of proficiency scales: insights from actual use

25

THANK YOU!!!

Contact details

Spiros PapageorgiouDept of Linguistics & English LanguageLancaster UniversityLA1 4YTUKs.papageorgiou@lancaster.ac.uk