Michael Roberts, Cathy Taylor, Veronika Timpe

Outline

1. Our task

2. Widgets

3. TBLT & TBLA

4. Approach to TBLA

5. Needs Analysis

6. Test design and purpose

7. The tasks and their assessment criteria

8. Limitations

9. References

2

Our taskDesign a task-based language test for a group of students who have done the first unit of the task-based course in practical English Widgets+ Appendix C: Video Scripts

-design decisions-Instrument(s)-tasks will the students have to do -judgment criteria-generalisability, task difficulty, and task variability

3

Widgetshttp://www.widgets-inc.com/teacher/tblt.php

“What makes Widgets truly different from other language teaching textbooks is its focus on creating a believable, “real-life” English-speaking environment in the classroom. Students imagine that they are new employees at an exciting international company, Widgets Incorporated, where they must work together in small groups to perform various practical, yet fun, tasks.”

4

Widgets (2)Initial reservations with the course:

Are the tasks truly authentic when they often centre round make-believe, weird products

We did not find these fantastical products either fun or motivating

Language is given before a task to be used in that task (not TBL)

There is little evidence of language being fed in when the need arises (this is TBL)

TBL is only one type of methodology for teaching language.

Shouldn’t it be presented with others to provide a balanced syllabus?

Is it desirable that a course book confines itself to one methodology?

5

TBLT & TBLA

“Task-based language assessment takes the task itself as the fundamental unit of analysis motivating item selection, test instrument construction, and the rating of task performance. Task-based assessment does not simply utilize the real-world task as a means for eliciting particular components of the language system, which are then measured or evaluated; instead, the construct of interest is performance of the task itself.”

Brown, Hudson Bonk and Norris

“Many traditional methodologies begin by teaching grammatical forms and then go on to set communicative activities in which they believe learners will be able to use these forms. The initial aim of TBT is to encourage learners to engage in meaning with the language resources they already have. This makes learners acutely aware of what they need to learn.”

Dave and Jane Willis

6

Approach to TBLA

7

1. Specify intended uses for the assessment: who, what, why, impact

2. Select and analyze key target tasks/features from needs analysis

3. Design tests and items: authentic elicitation, evaluation conditions

4. Determine real-world criteria for rating task performance qualities

5. Pilot-test and revise instruments and procedures (raters/rating)

6. Evaluate validity in terms of intended uses, and especially impact

on teaching and learning.

(Long & Norris 2000)

1) Intended assessment use

8

Who?Test takers:-12-40 Asian students -between 16-24 years old

“Stake holders” :-Teachers-parents

What?Similar tasks that the students have been exposed to/ dealt with in the teaching

Impact?-Washback effect-Low stakes

Why?The purpose of the test is to gather information about the student learning, i.e. whether the students, after having completed the first unit in Widgets, are able to perform these tasks and how language is used among them. Moreover, the assessment might support further teaching and learning.

2) Key target tasks & features from NA

9

Can do statements from Widgets

10

2) Key target tasks & features from NA (2)Objectives from Widgets

3) & 4) Test design & tasks

11

Importance of task assessment being based on tasks known or familiar to the students

Achievement test

Transfer of the tasks from the context of the company (as in the unit) into a different contextual framework: biannual trade fair

assessment spin-off to unit 1

Tasks and their assessment criteriaTask 1 – speaking (role play):

Task title: Catching up over coffeeTheme: ChatLevel: intermediateTime frame: 3-5 minutes per pairResponds to: water cooler chatCommunicative Mode: Role Play

Description of task:As an employee of Widgets Inc. you are attending a biannual trade fair in New York where you meet people from other companies. You are acquainted with most of them. On the first day during a conference break you meet a former colleague of yours who now works at another Widgets subsidiary. You two enjoy the chance to catch up socially over coffee. In pairs exchange information about events in your lives over the last 12 months. The prompt cards provided will help you find topics to talk about (e.g. work, holidays etc.).

12

13

Tasks and their assessment criteria (2)Task 1 – assessment criteria

a) Task-dependent criteria (Norris et al 2002)

appropriateness of questions

ability to produce follow-up questions that show the speaker A has listened to speaker B

ability to respond to follow-up questions appropriately rather than beginning a new topic

ability to maintain dialogue and begin a new topic when necessary

b) Task-independent criteria

appropriateness of vocabulary and structure for intermediate

discourse markers

turn taking elements

rejoinders

14

Tasks and their assessment criteria (3)

Task 1 – Scoring

5-0 scale

Reason: reflects the self-assessment task at the end of stage 1

Tasks and their assessment criteria (4)Task 2.1 – listening and note taking:

Task title: Competing productsTheme: Advertisement Level: IntermediateTime frame: depends on the length of recordings (approx. 6-7 min)Responds to: “Water cooler chat” & “That’s my department”

Communicative Mode: Aural reception/ information gap filling

Description of task:As a conference participant you decide to listen to the presentation of three products from a competitor. You and your company have heard some things of the products before, but are still missing more detailed information about the products. You listen to the presentation to find out more details about the product and to tell your superiors more about the products later on. Therefore, you will listen to the presentations twice. While listening to the presentation for the first time, take notes on the form provided. When listening a second time make notes that you will need to tell your boss more about the competitive products.

15

16

Form for information gap task 2.1

17

Tasks and their assessment criteria (5)Task 2.1 – assessment criteria

a) Task-dependent criteria (Norris et al. 2002)

-Completeness of information

-Listening for detail/ keyword/ gist

-I can take short notes in English

b) Task-independent criteria

-Alternative spellings

18

Tasks and their assessment criteria (6)

Task 2.1 – Scoring

a) Information gap

-Point for each blank

-Cutoff: 3 correct

b) Note taking

- Will be taken into consideration in the context of the presentation

Task 2.1 = scaffolding for the following presentational task

Tasks and their assessment criteria (7)Task 2.2 - presentation:

Task title: Product presentationTheme: Marketing Level: intermediateTime frame: Preparation 20 min; task 3-5 minResponds to: “Sell, sell, sell”Communicative Mode: Presentational

Description of task:You have returned to your company site from the conference. Your superiors are eager to know what you learned and which of the products you consider to be the most likely competitor. Go back to your notes from the listening task and prepare a short presentation for your group on the product you consider to be the most competitive one for Widgets.

19

20

Tasks and their assessment criteria (8)Task 2.2 – assessment criteria

a) Task-dependent criteria (Norris et al. 2002)

-speaks smoothly without any pauses

-structure of the presentation: introduction, description, good and bad points, why it's a

competitor, conclusion

-quality of delivery - longer turns, lack of hesitation, clarity of pronunciation

- does the presentation content reflect a success of the listening

-b) Task-independent criteria

-signpostings

-should we include body language/gestures?

21

Tasks and their assessment criteria (9)

Task 3 – Scoring

-Scale numerical 5-0

-Descriptors such as from excellent to not sufficient

Limitations

22

General limitations:

Not Constructed the materials

Interlocutor frame for assessment

Outline of task, rather than creation of tasks

Descriptors on rating scales need to be considered

Concerns expressed in the literature no easy answers

Limitations based on Widgets:

No clear language objectives outlined in Can do statements

Level of learners is not well-defined

Can Do statements are very general

suggestion to have an overall written report on the entire assessment series

ReferencesElder, C., Iwashita, N. & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: what does the test-taker have to offer? Language Testing, 19 (4), 347 – 348.

Long, M. H., & Norris, J. M. (2000). Task-based teaching and assessment. In M. Byram (Ed.), Encyclopedia of language teaching (pp. 597-603). London: Routledge.

Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities andtask difficulty in task-based second language performance assessment. LanguageTesting, 19(4), 395-418.

Wigglesworth, G. (2008). Task and performance based assessment. In S. Shohamy & N. H. Hornberger(Eds.), Encyclopedia of language and education (2nd Ed.), Volume 7: Language testing and assessment (pp.111-122). Springer Science+Business Media.

Willis, D & Willis, J. (2007) Doing task-based Teaching, Oxford

http://flenj.org/CAPS/?page=parent

23