Standardisation of judgements: Standardisation of judgements: reading and writingreading and writing

Ülle TürkUniversity of Tartu/Estonian Defence Forces

ulle.turk@ut.ee/ylle.tyrk@mil.ee

EUROPOS SĄJUNGA

ReadingReading

EUROPOS SĄJUNGA

30-31 March 2007 3

Training with standardised samples Training with standardised samples for readingfor reading

• Phase 1: Illustration – a first assessment of the level of one text and its corresponding tasks and items– At what CEFR level can a test taker already answer the

following item correctly?• Phase 2: Controlled practice – different texts with

their corresponding tasks and items are assessed under the guidance of the co-ordinator

• Phase 3: Individual assessment of the remaining items in chunks of 4-6 itemsThe session continues until a spread of results of one and a half levels is achieved.

EUROPOS SĄJUNGA

30-31 March 2007 4

Benchmarking local samples of Benchmarking local samples of readingreading

• Individual rating followed by (pair) discussion leading to group consensus

• Discussion of spread across individual ratings and iteration until suitable agreement (spread equal to one and a half levels) is reachedRatings of the CEFR standardised samples should be analysed statistically in order to

a) Confirm the relationship to the levels andb) Calculate intra-rater reliability (consistency) and inter-

rater reliability

EUROPOS SĄJUNGA

30-31 March 2007 5

Task 4: Items 1Task 4: Items 1––44

• Text Characteristics: – Text source: magazine – Authenticity: authentic – Discourse type: mainly argumentative– Domain: public – Topic: other/politics – Nature of content: fairly extensive abstract content – Text length 350 words – Vocabulary: extended vocabulary– Grammar: wide range of complex structures – Text likely to be comprehensible by learner at CEF level: C1

EUROPOS SĄJUNGA

30-31 March 2007 6

Items 1 & 2Items 1 & 2

• Item 1– Item type: Selected

response/multiple choice

– Operations: Infer implicit main idea/gist

– Item level estimated: A2/B1

– Key: B

– Per cent correct: 90.5%

– Discrimination (item/total correlation): .31

• Item 2– Item type: Selected

response/multiple choice

– Operations: Make explicit inferences about the writer’s mood

– Item level estimated: C2

– Key: B

– Per cent correct: 36.1 %

– Discrimination (item/total correlation): .45

EUROPOS SĄJUNGA

30-31 March 2007 7

Items Items 33 & & 44

• Item 3– Item type: Selected

response/multiple choice

– Operations: Make inferences about implicit main idea/gist

– Item level estimated: C2

– Key: B

– Per cent correct: 43.3 %

– Discrimination (item/total correlation): .20

• Item 4– Item type: Selected

response/multiple choice

– Operations: Make inferences about implicit detail

– Item level estimated: B1

– Key: C

– Per cent correct: 83.6 %

– Discrimination (item/total correlation): .43

EUROPOS SĄJUNGA

30-31 March 2007 8

Task 5: Items 5Task 5: Items 5––88

• Text Characteristics: – Text source: magazine – Authenticity: authentic – Discourse type: mainly expository – Domain: public/personal – Topic: other/contemporary life– Nature of content: mainly abstract content – Text length 418 words – Vocabulary: extended vocabulary – Grammar: mainly complex structures – Text likely to be comprehensible by learner at CEF level: C1

EUROPOS SĄJUNGA

30-31 March 2007 9

Items Items 55 & & 66

• Item 5– Item type: Selected

response/multiple choice

– Operations: Infer implicit detail

– Item level estimated: B2

– Key: B

– Per cent correct: 65,9 %

– Discrimination (item/total correlation): .52

• Item 6– Item type: Selected

response/multiple choice

– Operations: Evaluate implicit opinion

– Item level estimated: C2

– Key: C

– Per cent correct: 49,2 %

– Discrimination (item/total correlation): .32

EUROPOS SĄJUNGA

30-31 March 2007 10

Items Items 77 & & 88

• Item 7– Item type: Selected

response/multiple choice

– Operations: Make inferences about implicit main idea/gist

– Item level estimated: C1

– Key: C

– Per cent correct: 52,1 %

– Discrimination (item/total correlation): .36

• Item 8– Item type: Selected

response/multiple choice

– Operations: Evaluate implicit attitude/conclusion

– Item level estimated: B2

– Key: A

– Per cent correct: 74,4%

– Discrimination (item/total correlation): .24

EUROPOS SĄJUNGA

WritingWriting

EUROPOS SĄJUNGA

30-31 March 2007 12

Training with standardised samples Training with standardised samples for writingfor writing

• Phase 1: illustration with about three standardised performances – 60 minutes

• Phase 2: controlled practice with 3–5 standardised performances – 60 minutes

• Phase 3: individual assessment of 3–5 standardised performances – 60 minutesThe session ends when the vast majority of participants agree on the level, with the spread not exceeding one and a half levels.

EUROPOS SĄJUNGA

30-31 March 2007 13

Benchmarking local samples of Benchmarking local samples of writingwriting

• Individual rating and group discussion of high, middle and low performances – 60 minutes

• Individual rating of about five more performances – 60 minutes

EUROPOS SĄJUNGA

30-31 March 2007 14

Sample taskSample task 1 1

• Your English Teacher has asked you to write a story.

• Your story must have the following title:

The most important day of my life

• Write your story on your answer sheet.

EUROPOS SĄJUNGA

30-31 March 2007 15

Example answerExample answer 1 1

The most important day of my life.

During a lifetime there is so many days you could call, the most important day of your life”. It could be the day you chose wich school you are going to, or what you want to work with the rest of your life. Another important day is when you get married, or you chose where to live. But most of all it must be a very important day when you give birth to a child. I think that changes everything you have been doing until then. Than you have to realice that somebody are more important than yourself.

EUROPOS SĄJUNGA

30-31 March 2007 16

Sample taskSample task 2 2This is an extract from a letter which you receive from the World Opinion Organisation.

We are carrying out an international survey on families around the world and the importance of family relationships. Please write us a report for our survey outlining the typical family situation in your own country and suggesting how you think the situation might change in the future. Write your report. Write approximately 250 words.

EUROPOS SĄJUNGA

30-31 March 2007 17

Sample taskSample task 3 3

You have read the extract below as part of a newspaper article on the loss of national and cultural identity. Readers were asked to send in their opinions. You decide to write a letter responding to the points raised and expressing your own views.

‘We are losing our national and cultural identity. Because of recent advances in technology and the easy availability and speed of air travel, different countries are communicating more often and are therefore becoming more and more alike. The same shopping malls and fast food outlets can be found almost everywhere. So can the same types of office blocks, motorways, TV programmes and even lifestyles. How can we maintain the traditions that make each nation unique?’

Write your letter. Do not write any postal addresses.

EUROPOS SĄJUNGA

30-31 March 2007 18

Sample taskSample task 3 3

• You recently entered a competition and have just received this letter from the organiser. Read the letter, on which you have made some notes. Then, using all the information in your notes, write a suitable reply.

• Write a letter of between 120 and 180 words in an appropriate style on the opposite page. Do not write any postal addresses.

EUROPOS SĄJUNGA

30-31 March 2007 19

EUROPOS SĄJUNGA

30-31 March 2007 20

SourcesSources

• Manual for relating language examinations to the Common European Framework of Reference for Languages – a preliminary pilot version: http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/Manual.html

• The CEFR Grid for Writing, v. 2.:• The Final Grid for analysing reading and listening

items: www.ling.lancs.ac.uk/cefgrid

EUROPOS SĄJUNGA