Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses

Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL

are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906

Assessing speaking skills

using human raters

in an Internet-based environment

Joanna Wrzesińska ETS Global

Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are

registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906

What we will be doing today

1. Introduction: Review the iBT Speaking section content

2. Discuss rater training

3. Explain and use the scoring guides and

benchmarks for Speaking to rate real test-taker

responses



Speaking



TOEFL iBT® Test

Speaking Section • Test takers respond to 6 tasks

• 2 independent tasks about familiar topics

– Responses based on personal experience

• 4 integrated tasks

– 2 Reading/Listening/Speaking tasks (1 academic, 1 campus-based)

– 2 Listening/Speaking tasks (1 academic, 1 campus-based)

– Responses based on what was read and heard

• Time: 20 minutes



TOEFL® Rater Training & Scoring

Apply Intensive ETS

Training Daily Calibration

Test

Monitoring by Scoring

Leader

Analyze Results

1 3 2 5 4



Speaking Scores

• Holistic scoring

• Test takers are rated on:

– Delivery

– Language use

– Topic development

• Ratings are 0 - 4

– Converted to a scaled score of 0 - 30



Speaking Benchmarks

• What is a Benchmark?

• Listen to the 4 Benchmarks for Speaking Task

#1 - Independent Speaking





Most test takers should be able to respond to this topic easily, although it is likely that

very few will limit their discussion to just “one way” their life will be different. Even test

takers with responses at the highest level may introduce more than one aspect of their

lives that they think will be different. These responses are still eligible for a 4 as long as

some detail and development is included in their discussion.

Lower level respondents may merely list a number of ways that their lives may be

changed without much development of any of them (“I will be married, have children, a

different job,” etc.); these types of responses will most likely not be eligible for a score of

4 for topic development.

Some candidates may misinterpret the prompt and talk about the world in general

(technology, etc.) without connecting it to their own experience, or misinterpret the prompt

in some other way (e.g., talking about ten years ago). Responses that misinterpret the

prompt generally will not be eligible for high marks. Consult your Scoring Leader

whenever you are unsure of the appropriate score when considering topic development.

Topic Notes



TOEFL iBT® Speaking Section

Benchmark speaking responses • Benchmarks – independent speaking

• Annotations provide narrative description of the level

• Each score represents a range of ability

– Level 4

– Level 3

– Level 2

– Level 1



Now Try Rating the Independent Task

– Listen to each response

– Use the Independent Speaking Rubric

– Give a rating from 1 to 4; take notes on rationale (how it fits the descriptor)

– Discuss your rationale in groups or pairs

Test taker number 1

Test taker number 2



Integrated Speaking Task • Task #4 - General/Specific

• A Reading/Listening/Speaking Academic Topic

• Consists of two sources: a short reading and a lecture, followed by a single question

– Test taker reads a paragraph (~100 words) about an academic topic

– Test taker hears an excerpt from a lecture illustrating the topic (~90 seconds)

– The test taker is asked to briefly combine and convey important information from the reading and lecture

• Note taking allowed

• Prep time: 30 seconds

• Response time: 60 seconds









Key Points

• Relict behavior is a behavior that an animal displays that once served a

purpose (for survival) but no longer serves that purpose.

• In the past, the (North American) pronghorn’s speed was critical for its

survival because its predator, the lion, also ran very fast.

• Lions are now extinct (in North America), but the pronghorn still displays the

relict behavior of running extremely fast even though that is no longer

required for survival (as it is so much faster than its current predators,

bobcats and coyotes).



Use the Integrated Speaking Rubric to

match the characteristics of each response

to a score.

– Test Taker 1

– Test Taker 2

Now Try Rating the Integrated

Task



Thank you!

[email protected]

Documents

Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses