Welcome!

The Grid - Reloaded Competence-based test design and

assessment

The theory: what we need for a...

Good test

Validity

Reliability

Feasibility

Authenticity

Validity

= The degree to which the test actually

measures what it intends to measure.

Reliability

= The consistency of results that a particular test produces.

Reliability

Test reliability Reliability of

test conditions

Student-related reliability

Scorer reliability

Feasibility

= is the test practical... ◦  to set up? ◦  to administer? ◦  to assess/mark? ◦  to finance?

Authenticity

àHow convincingly does the test simulate “real-world” tasks?

� Can be increased through: ◦ Contextualisation �  “Authentic” topics, situations �  Linked sentences, not isolated ones à thematic organisation (story, episodes...)

◦  Layout and instructions of test tasks

Keep in mind:

The importance of specifying: -  Form

-  review, story, letter...

-  Audience -  familiarity, distance

-  Topic -  Purpose

-  entertain, persuade, describe...

Any remaining questions?

Good, because now it’s YOUR turn...

What you need to do now:

1.  Use the marking grid you are given to assess and mark the three student productions.

2.  Use the Think Aloud Protocol to record any reflections, justifications... that underpin your decision(s).

3.  Compare and discuss your findings with the other members of your group.

Conclusion: Why work with marking grids?

� Criterion-referenced assessment ◦  Systematic, nuanced ◦ Qualitative and quantitative considerations ◦  Accountability

� Avoiding dangers of ‘gut-feeling’ assessment �  Increasing scorer reliability �  Standardising assessment procedure � Chance for focused, detailed feedback � A positive look at the “bigger picture”

BAND 0 BAND 1 BAND

2 BAND 3 BAND 4 BAND 5

CONTENT •  Task achievement  •  Effect on reader  •  Format (if required)  

No

rate

able

lang

uage

. Tot

ally

inco

mpr

ehen

sibl

e. T

otal

ly ir

rele

vant

Only 1/3 of the content elements dealt with and / or hardly any content elements dealt with successfully. Message only partly communicated to the reader and / or considerable effort required on behalf of the reader. Format only partly respected.

Feat

ures

of b

ands

1 a

nd 3

. S

erio

us e

ffort

by re

ader

requ

ired.

Only 2/3 of the content elements dealt with: message only partly communicated to the reader; and /or not all content elements dealt with successfully. Support data might be missing or there might be irrelevant detail. Some effort required of the reader. Awareness of format, but occasional inappropriacies.

Feat

ures

of b

ands

3 a

nd 5

. Li

ttle

effo

rt by

read

er re

quire

d.

All content elements covered appropriately: message clearly communicated to reader. The desired effect on reader is fully achieved. Overall, no effort required of the reader. The format is appropriate

ORGANISATION: COHERENCE AND COHESION •  Logic  •  Fluency  •  Control of linking devices,

referencing...  •  Paragraphing  •  Punctuation  

Response may contain some inconsistencies. Only short simple sentences. At times limited control of simple sentences. The sentences are simply listed rather than connected and presented as a text.

Overall coherent response. Mostly good control of simple sentences. Complex sentence forms attempted, but they tend to be less accurate than simple sentences. Use of simple linking devices such as ‘and’, ‘or’, ‘so’, ‘but’ and ‘because’. Information presented with some organisation.

Coherent response. Good control of simple sentences and in general also of complex sentences. Coherence is achieved by use of simple linking devices. Response is well organised.

LEXIS •  Appropriacy  •  Range  •  Accuracy  

Limited range which is minimally adequate for the task and which may lead to repetition. Limited control of spelling and / or word formation.

In general, appropriate and adequate range for the task. Errors, which do not reduce communication and are mainly due to inattention or risk taking.

Confident and ambitious use of a wide range of vocabulary within the task set. Few minor errors which are non-impeding, perhaps due to ambitious attempts at more complex language.

GRAMMATICAL STRUCTURES •  Appropriacy  •  Range  •  Accuracy  

Limited range of structures. Frequent grammatical errors and faulty punctuation, which cause difficulty in terms of communication.

Sufficient range of structures for the task. Errors, which do not reduce communication and are mainly due to inattention or risk taking.

Confident and ambitious use of a wide range of structures within the task set. Few minor errors which are non-impeding, perhaps due to ambitious attempts at more complex language.

BAND 0 BAND 1 BAND 2 BAND 3 BAND 4 BAND 5

CONTENT •  Task response  •  Relevance  •  Effect on listener  

No

rate

able

lang

uage

. Tot

ally

inco

mpr

ehen

sibl

e. T

otal

ly ir

rele

vant

.

Meaning only conveyed in very familiar everyday situations. Response is only partly relevant. Serious effort on behalf of the listener required.

Feat

ures

of b

ands

1 a

nd 3

. S

erio

us e

ffort

by li

sten

er re

quire

d.

Communication handled in everyday situations. In general, response is relevant. Generally meaning is conveyed successfully, occasional effort

Feat

ures

of b

ands

3 a

nd 5

. Li

ttle

effo

rt by

list

ener

requ

ired.

Communication on familiar topics is handled well. Communication on unfamiliar topics is limited. Contributions are relevant despite some repetition. Meaning is conveyed successfully

PRONUNCIATION AND DISCOURSE MANAGEMENT •  Pronunciation •  Fluency (speech rate and

continuity) •  Effort to link ideas and language

so as to form a coherent, connected speech

•  Prompting and support •  Interaction

Speech is mostly intelligible, despite limited control of phonological features. Considerable effort on behalf of the listener required. Noticeable pauses and slow speech, frequent repetition and self-correction. Very short basic utterances, which are juxtaposed rather than connected or linked through repetitious use of simple connectives. Requires prompting and support.

Generally intelligible though mispronunciation may occasionally cause momentary strain for the listener. Produces simple speech fluently. Usually maintains flow of speech but uses repetition, self-correction and / or slow speech to keep going. May over-use or under-use certain connectives and discourse markers. Requires little prompting and support.

Generally intelligible. Intonation is generally appropriate. L1 is perceptible but does not interfere with intelligibility. Produces extended stretches of language but may lose coherence at times due to occasional repetition, self-correction or hesitation. Uses connectives and discourse markers but not always appropriately. Requires no prompting and support. Initiates and responds appropriately.

LEXIS •  Appropriacy  •  Range  •  Accuracy  •  Paraphrase  

Limited range of vocabulary which is minimally adequate for the task and which may lead to repetition. Paraphrase rarely attempted.

In general, adequate range of vocabulary for familiar topics. Attempt to vary expressions, but with inaccuracy. Paraphrase attempted, but with mixed success.

Has a wide enough range of vocabulary to discuss familiar topics at length and make meaning clear in spite of some inappropriacies. Unfamiliar topics may provoke inaccuracies and errors. Generally paraphrases successfully.

GRAMMATICAL STRUCTURES •  Appropriacy  •  Range  •  Accuracy  

Only a limited range of structures is used. Limited control of basic forms and sentences. Subordinate structures are rare and tend to lack accuracy. Frequent grammatical errors.

In general, adequate range for familiar topics. Good degree of control of simple grammatical forms and sentences. Both simple and complex forms are used. Minor and occasional errors.

Uses simple and complex structures, but with limited flexibility. May make mistakes with complex structures, though these rarely cause comprehension problems.

Thank you for attending!