Beata Schmid & Alexandra Bianco: Assessing oral communication skills

Eaquals International Conference, 16 – 18 April 2015

Assessing Oral Communication Skills

Alexandra Bianco and Beata Schmid

EF International Language Centres, Malta and Boston

Malaga, April 18, 2015

www.eaquals.org

Outline of presentation

• Background

• Oral activity types in the classroom

• Scope

• Idea share

• Other ideas

• Practical issues

• Don’t forget!

• References

• Q & A

2 Eaquals International Conference, 16 – 18 April 2015

What this is and what it’s not

• This is about informal

assessment in the classroom, on

a daily basis

• “Assessment on the fly”

• “Informal formal”

• It is not about formal oral

assessment


Who is here today

• Where do you teach? Intensive or extensive courses?

• How do you teach? What would you call your methodology or approach?

• What does your classroom look like? What is happening in your classroom in a normal lesson?


… and the million dollar question:

How do you assess your students’

oral activities in the classroom on

an on-going basis?


Background to EF & this project • Intensive, year-round, continuous enrolments

• 30 English language schools worldwide; 12 others

• Task-based, communicative classrooms

• “Endless, aimless presentations”

• Feedback at EAQUALS inspection, EF Malta:

• “In relation to the productive skills, standardization with teaching staff of performance-based assessment with CEFR-derived criteria is recommended. “

• What is the message we are sending students?

• Continuous development towards an environment where students take increasing responsibility for their learning (autonomous learning)

• We are not going to address formal assessment as in oral placement tests or formal speaking exams.


Oral activity types in the classroom

With your neighbor, brainstorm what different types of oral activities you might do in the classroom.

• Pair/group work • E.g., interviews, discussions; information gap

activities, etc.

• Presentations

• Skits/role play

• Simulations

• …


And how will you assess these activities?

Which statements do you agree with?

Speaking assessments …

• … should be done formally every week

• … should be done regularly but

informally

• … should be done individually

• … should involve student input

• … should…

Eaquals International Conference, 16 – 18 April 2015 8

What is assessment?

• Measuring proficiency & achievement

• Diagnosing capabilities and finding gaps

• Helping us evaluate whether lessons have

been effective

• Practicality

• Reliability } features of good assessment

• Validity


Oral assessment is difficult…

• Among the language skills we assess, it’s widely recognized to be the most difficult one to assess.

• What exactly are we assessing? • Pronunciation? Spoken grammar?

Spoken vocabulary? Speech acts? Accuracy? Fluency?

• Global or detailed assessment?

• Individual students or groups?

• Subjective or objective?


Let’s take a step back

• What is your proficiency scale?

• Many use the Common European

Frame of Reference (CEFR); so

do we.

• Let’s look at the CEFR Oral

Assessment Criteria.


CEFR, e.g., B1

Global Oral Assessment Scale

Relates comprehensibly the main points he/she wants to make.

Can keep going comprehensibly, even though pausing for grammatical and lexical planning and repair may be very evident. Can link discrete, simple elements into a connected, sequence to give straightforward descriptions on a variety of familiar subjects within his/her field of interest. Reasonably accurate use of main repertoire asso-ciated with more predictable situations.

• Specifically, the CEFR assesses Range, Accuracy, Fluency, Interaction and Coherence, like this (for B1):


CEFR Oral Assessment Criteria Grid (e.g., B1): qualitative aspects of spoken language


Range Accuracy Fluency Interaction Coherence

Has enough

language to get

by, with sufficient

vocabulary to

express

him/herself with

some hesitation

and circum-

locutions on

topics such as

family, hobbies

and interests,

work, travel, and

current events.

Uses

reasonably

accurately a

repertoire of

frequently used

“routines” and

patterns asso-

ciated with

more

predictable

situations.

Can keep

going

comprehensibl

y even though

pausing for

grammatical

and lexical

planning and

repair is very

evident,

especially in

longer

stretches of

free

production.

Can initiate,

maintain and

close simple

face-to-face

conversation

on topics that

are familiar or

of personal

interest. Can

repeat back

part of what

someone has

said to confirm

mutual

understanding.

Can link a

series of

shorter,

discrete

simple

elements into

a connected,

linear

sequence of

points.

Let’s try it out. (Handout)

• Scenario 1

• Long-term students

in an intensive

program. The task-

based syllabus

contains lots of

group

presentations.

• Scenario 2

• Short-term students

(2-3 weeks) in an

intensive program.

Lots of interviews

and role play.


Let’s try it out. (Handout)

Scenario 3

• Long-term students

in an extensive

program. The task-

based syllabus

contains lots of

individual

presentations.

Scenario 4

• Students in an

extensive program

with casual

attendance. Lots of

speaking activities

in class.

• What is your

specific scenario?


Group work

• Scenario, specific speaking task

• Possible oral assessment format


Here’s an actual task from one of our coursebooks

• This B1 unit is about manners and the task is to create and perform skits on cultural mistakes.

• Students are asked to choose a culture they know well and to brainstorm bad manners or cultural mistakes in the following situations. Choose one.

• On public transportation

• In a restaurant

• At a party

• How can we assess students?


Let’s specify the criteria:



Has enough

language to get

by, with sufficient

vocabulary to

express

him/herself with

some hesitation

and circum-

locutions on

topics such

_____________

___________________________________________________________________________

Uses

reasonably

accurately a

repertoire of

frequently

used “routines”

and patterns

associated

with more

predictable

situations, for

example,

____________

____________

____________

____________

____

Can keep going

comprehensibly

even though

pausing for

grammatical and

lexical planning

and repair is

very evident,

especially in

longer stretches

of free

production.

Can initiate,

maintain and

close simple

face-to-face

conversation on

topics that are

familiar or of

personal

interest. Can re-

peat back part of

what someone

has said to con-

firm mutual

understanding.

Can link a

series of

shorter,

discrete simple

elements into

a connected,

linear

sequence of

points, for

example,

___________

___________

___________

___________

___________

___________

___________

___________

___________

_____

For example



Vocabulary and

phrases related

to politeness

(and to identify

impoliteness),

such as small

talk phrases,

apologising,

thanking

someone,

asking

permission,

asking for

something

Uses

reasonably

accurately a

repertoire of

frequently

used phrases

such as the

ones describe

under

“Range:” e.g.,

Excuse me, I

am sorry,

but… , Why

don’t you… , I

wonder if…

Can keep

going

comprehensi

bly even

though

pausing for

grammatical

and lexical

planning and

repair is very

evident,

especially in

longer

stretches of

free

production.

Can initiate,

maintain and close

simple face-to-face

conversation on

topics that are

familiar or of

personal interest.

Can repeat back

part of what

someone has said

to confirm mutual

understanding

regarding

good/bad

behaviour.

Can link a

series of

shorter,

discrete

simple

elements into

a connected,

linear

sequence of

points in order

to describe

what

happened in

the skit.

Possible format


A. Information/topic: Circle one

1. The group presented interesting information. 3 2 1 0

Comments/examples:

B. Organization

1. The skit was well organised and presented information in a

logical way.

3 2 1 0

2. The information was relevant. 3 2 1 0

Comments/examples:


C. Preparation

1. The presenters were well prepared with good information and

visual aids.

3 2 1 0

2. Name(s) of specific students that stood out _________________ 3 2 1 0

Comments/examples:

D. Presentation

1. The skit was engaging and well received by the class 3 2 1 0

2. The presenters spoke clearly and fluently. 3 2 1 0

3. The presenters used good gestures and eye contact. 3 2 1 0

Comments/examples:

E. Interaction and involvement

1. The presenters got us involved in the presentation. 3 2 1 0

2. We learned something new (please say what in the comments). 3 2 1 0

Comments/examples:

What’s missing here?

• Language!

• CEFR criteria!

• But teachers liked it!


Group Observation Grid for Manners Skit Stage: B1 Task: Perform a skit on cultural mistakes Aims: Make polite requests with be supposed to; discuss how to handle impoliteness ; listen to advice about good behaviour


Focus of

assessment Comments

Student Name

Comments

Student Name

Comments

Student Name

Comments

Student Name

Range:

small talk phrases,

apologizing

Accuracy:

Be supposed to

Fluency:

Comprehensive,

fluent speech

Interaction:

Natural,

conversation

Coherence:

Linked speech

Overall Assessment

Group Observation Grid for Manners Skit Stage: B1 Task: Perform a skit on cultural mistakes Aims: Make polite requests with be supposed to; discuss how to handle impoliteness ; listen to advice about good behaviour SAMPLE COMPLETED FORM, Group 3

Eaquals International Conference, 16 – 18 April 2015

Focus of

assessment Comments

Student A Comments

Student B

Comments

Student C

Comments

Student D

Range:

small talk phrases,

apologizing

What are you

doing? What am I

supposed to do?

P

P

Accuracy:

Be supposed to P P

P

We are

supposed to

giving this X

P

Fluency:

Comprehensive,

fluent speech

What this

noise?

P

P

Very fluent

Interaction:

Natural,

conversation

P

P

P

It means the back

side of the women,

not polite for me

Coherence:

Linked speech

P

P

P

P

Overall Assessment A- A B B+

Other solutions

• Some teachers assigned numbers, like this:

• 1-2: minimal

• 3-4: limited

• 5-6: competent

• 7-8: proficient

• 9-10 exceptional

Similar to another scheme we have.

• Others used letters:

• F = fair, 1 point

• G = good, 2 points

• VG = very good, 3 points

• Overall assessment out of 15 points

• Typical scores: 12/15,

• 11/15

Or other number scores.


Another format: Interview Assessment Grid


Criteria Yes

Student A/Student B

Sometimes

Student A/Student B

Not very often

Student A/Student B

The students

asked each other

q’s and answered them.

The students

listened to each other.

The students

responded

appropriately.

There was

appropriate turn-

taking.

Students used the

specific language

taught previously,

e.g.,

(examples)

Another format: Interview Assessment Grid Aims: Using correct intonation, body language and phrases for commiserating


Criteria Yes

Paulette & Roberta

Sometimes

Student A/Student B

Not very often

Student A/Student B

The students

asked each other

q’s and answered them.

P

The students

listened to each other.

P

The students

responded

appropriately.

Great intonation.

Excellent body

language

There was

appropriate turn-

taking.

P

Students used the

specific language

taught previously,

e.g.,

(examples)

Really? OMG That’s amazing! Gestures, facial expressions, eye contact

Feedback

Jeannie


I would have preferred brief descriptions of the

items being assessed, e.g., range, accuracy,

etc. instead of being supplied separately [by

the teacher?]. I would provide more spaces for

positive and negative comments. The grid is a

good guide as the focus areas of assessment

are clearly indicated and it’s user friendly, too.

More feedback


It made me think about

the CEFR criteria, but

it’s a bit difficult to

use. It’s OK for the

teacher’s assessment,

but not for peer

assessment – it’s too

complicated.

I wish that suggested

assessment schemes

were included with the

activities in the

textbooks.

What we learnt

• Teachers appreciated having a format for evaluating students’ oral performance in class, but

• Teachers needed the forms to be simpler to use

• Teachers were assessing on a gut feeling, sometimes measured by the competence of the other students in the class.


What we did in Malta

• Started with the general presentation to motivate the need for assessment of spoken activities, based on the Eaquals inspection.

• A bit daunting, so we …

• … broke it down into weekly tasks

• … tried different assessment schemes every week

• … collected feedback weekly

• … encouraged teachers to come up with their own format – in a workshop


More findings

• Teachers wanted ONE or TWO

formats for ALL speaking

activities – KISS

• So, here’s our goal:

• One format for most speaking

activities, and

• A separate form for presentations


Speaking Performance Assessment Form


Week Aim Student Comment

Teacher Remarks PR* VC &

GR**

SP &

SI***

*PR = Pronunciation: Grading based on CEFR criteria Accuracy, Fluency & Coherence

**VC & GR = Vocabulary & Grammar: Grading based on CEFR Criteria Range & Accuracy

***SP & SI = Spoken Production & Spoken Interaction: Grading based on CEFR Criteria Range,

Fluency, Accuracy & Interaction

ALL GRADING CRITERIA TIE IN WITH THE EF Efekta Progress Tracker

Speaking Performance Assessment Rubric


B1 Grammar and Vocabulary

(CEFR: Range & Accuracy) Discourse Management (Interaction & coherence)

Pronunciation (Accuracy & Fluency)

Interactive

Communicaiton (interaction & coherence)

9-10

Shows a good degree of control of

simple grammatical forms and

attempts some complex

grammatical forms

Produces extended stretches of

language despite some hesitation Is intelligible

Initiates and responds

appropriately

Contributions are relevant despite

some repetition Intonation is generally

appropriate Maintains and develops the

interaction and negotiates

towards an outcome with

very little support

Uses a range of appropriate

vocabulary to give and exchange

views on familiar topics Uses a range of cohesive devices

Sentence and word stress is generally accurately placed

Individual sounds are generally

articulated clearly

7-8 Performance shares features of Bands 3 & 5

5-6

Shows a good degree of control of

simple grammatical forms

Produces responses which are

extended beyond short phrases,

despite hesitation Is mostly intelligible and has

some control of phonological

features at both utterance and

word levels

Initiates and responds

appropriately

Uses a range of appropriate

vocabulary when talking about

familiar topics

Contributions are mostly relevant, but there may be some repetition

Keeps the interaction going

with very little prompting

and support Uses basic cohesive devices

3-4 Performance shares features of Bands 1 & 3

1-2

Shows sufficient control of simple

grammatical forms

Produces responses which are

characterised by short and frequent

hesitation Is mostly intelligible, despite

limited control of phonological

features

Maintains simple

exchanges, despite some

difficulty

Uses a limited range of appropriate

vocabulary to talk about familiar

topics

Repeats information or digresses

from the topic Requires prompting and

support

0 Performance below Band 1

For example, the rubric for speaking assessment for Cambridge ESOL

B1, PET.

How to use these forms


Where are we going from here?

• It’s a work in progress

• We plan to revisit the forms again before the summer: “standardization with teaching staff of performance-based assessment”

• Train summer teachers on the use of the forms

• Highlight units in our textbooks and workshop activities that lend themselves to this kind of oral assessment


After the summer….

Investigate other types of oral assessment, e.g.,

• More student involvement/self assessment

Review communicative tasks in the textbooks.

• Incorporate assessment schemes into next edition of EF textbooks


References

• Brown, J.D., ed., New Ways of Classroom Assessment, TESOL, 2013.

• Genesee, F. & Upshur, J.A., Classroom-based Evaluation

in Second Language Education. Cambridge University Press,

1996/2001

• Luoma, S. Assessing Speaking, Cambridge University Press, 2004.

Information on the CEFR:

• www.coe.int/lang (Common European Frame of Reference)

• relex.ecml.at/LinkClick.aspx?fileticket=%2Fx7%2BuhDqXBM%3D& for CEFR Oral Assessment Grids; last accessed August 20, 2014.


http://www.coe.int/lang

Final thought

Genesee, F. & Upshur, J.A., Classroom-based Evaluation

in Second Language Education. 1996/2001.

Cambridge University Press.


Evaluation is a process that

results in decisions about

instruction, students, or both.

Thank you!
