Computer-mediated Oral Assessment and Its Implications for Teaching

Speaking to Thai Students

Computer-mediated Oral Assessment and Its Implications for Teaching

Speaking to Thai Students

Kanchana Prapphal and Prakaikaew Opanon-amata

Chulalongkorn University Language Institute

The Computerized Oral ProficiencyTest

(COPI)

The Computerized Oral ProficiencyTest

(COPI)

• OPI (Oral Proficiency Interview)• SOPI (Simulated Oral Proficiency

Interview, tape-mediated)• COPI (Computerized Oral Proficiency

Test)

• Developed by the Center for Applied Linguistics

Characteristics of COPICharacteristics of COPI

• Non-linear• Different tasks • Computer-mediated • Easier feedback for examinees• Electronic storage of examinees’

performances • Electronic storage of rating data

(Malabonga et al., 2000)

Advantages of COPIAdvantages of COPI

• Time saving• Automatic computation of score• Report generation• Data-saving in electronic form• Efficient• Easy-to-use format• Flexible for raters

• Chula students should be able to

• *communicate in English

• *use the computer for their academic work as well as for their future career

Chula ContextChula Context

Objectives of the StudyObjectives of the Study

1. Assess the students’ speaking

ability by using the computer

2. Investigate the students’

communication strategies

Research QuestionsResearch Questions

1. Can computer-mediated oral assessment (CU COA) assess students’ oral proficiency?

2. What communication strategies do the students use in taking the CU-COA test?

SubjectsSubjects

• 30 first year students (16 male and 14 female) from the Faculties of Science, Education and Political Science

• They studied English for 8 years.

• Their equated TOEFL scores ranged from 370-640.

InstrumentInstrument

• The Chulalongkorn University Computer-mediated Oral Assessment (CU-COA)

• There are 4 sections:

- Introduction

- Section 1 (Give personal information.)

- Section 2 (Give information about familiar topics.)

- Section 3 (Describe future plans.)• Rating scale: 6 levels

Rating ScaleRating Scale

• Level 6 : Excellent speaker of English• Level 5 : Very good speaker of English• Level 4 : Good speaker of English• Level 3 : Fair speaker of English• Level 2 : Poor speaker of English• Level 1 : Very poor speaker of English

Descriptors of the ScaleDescriptors of the Scale

• Example :

Level 6 : -employs native-like prosody

-uses complex grammatical structures

- rarely makes a mistake

- links ideas coherently

Data CollectionData Collection

• October-November 2005

• 7 raters (28-59 years old)

• Rater training before the test administration

Data AnalysisData Analysis

Correlation coefficients

among ratersThey range from .514 to .884.

• All correlation coefficients are significant at the .01 level.

• One rater is the least reliable. (lacking the computer skills)

ResultsResults

• Grand mean = 3.198

• SD = .918

• Correlation with CU-TEP = .879

LevelsNo. of

Students

6 -

5 4

4 5

3 15

2 6

1 -

ConclusionConclusion

Research Question 1:

“Can CU-COA assess students’ oral proficiency?

The CU-COA can assess their oral ability and can classify them into different levels according to their proficiency levels.

Research Question 2Research Question 2

What communication strategies do the students use in taking the CU-COA

test ?

Communication StrategiesCommunication Strategies

Nakatani(2005), Dornyei & Scott(1997) and Bialystok (1990)

•Achievement or compensatory strategies

•Reduction or avoidance strategies

Achievement StrategiesAchievement Strategies

“Good learner” behaviors

Use various ways to reach the original goal by means of whatever resources are available

Avoidance StrategiesAvoidance Strategies

“Low-proficiency learner’s behaviors”

Avoid solving a communication problem and give up on conveying their message

Analysis of Speech ProductionAnalysis of Speech Production

• Four recorded oral tests selected

• Six communication strategies (Nakatani 2006)

1. Social Affective Strategies1. Social Affective Strategies

•Try to control their own anxiety•Try to enjoy the process of oral communication

•Are willing to encourage themselves to use English

•Risk making mistakes

1. Social Affective Strategies1. Social Affective Strategies

Levels Number of Words

Produced

Periods of Silence

5 62 None

4 49 15 seconds

3 35 22 seconds

2 12 40 seconds

Utterances of the level 5 student

“I think I have a lot of friends because I’m a friendly girl and I’m very lively.But I

have one best friend. Her name is Prae. We knew each other from 11th grade from high school. She was very nice and very

sincere and we still keep in touch now although she’s in another college. But

she will remain my best friend.”

1. Social Affective Strategies1. Social Affective Strategies

Utterances of the level 2 student

1. Social Affective Strategies1. Social Affective Strategies

“……..Yes, I have a lot of friends……..(A long pause of 40 seconds)………... Korawit is my best friend……(Silence until time’s up)

2. Message Abandonment Strategies2. Message Abandonment Strategies

• Give up their attempt to communicate

• Leave the message unfinished

2. Message Abandonment Strategies2. Message Abandonment Strategies

Levels Number of Words

Produced

Periods of Silence

5 62 None

4 49 15 seconds

3 35 22 seconds

2 12 40 seconds

3. Fluency-oriented Strategies3. Fluency-oriented Strategies

• Pay attention to the pronunciation

• Pay attention to clarity of their speech

• Try to make the speech flow

3. Fluency-oriented Strategies3. Fluency-oriented Strategies

Levels Number of Words

Produced

Number of Unclearly

Pronounced Words

5 83 None

4 38 7

3 35 (Mostly repeated words)

5

2 17 2

4. Accuracy-oriented Strategies4. Accuracy-oriented Strategies

• Desire to speak English accurately

• Pay attention to forms

• Seek grammatical accuracy by self-correcting

4. Accuracy-oriented Strategies4. Accuracy-oriented Strategies

Levels Number of Words

Produced

Number of Attempts/Failur

e for self-correction

5 230 1 attempt

4 135 4 failures

3 128 5 failures

2 20 2 failures

4. Accuracy-oriented Strategies4. Accuracy-oriented Strategies

Utterances of the level 5 student

“Well, now , I’m studying to be a computer programmer

at the Faculty of Science in the field of computer science at Chulalongkorn University. So, definitely, I want to be a

programmer in the future. But after I graduate from Chulalongkorn University, I might probably go for a

master’s degree, I suppose.(Continuing with correct English……….And then I would think about ….studying

more…maybe at MIT. That’s where I want to go although my Dad graduated from Harvard. Maybe he wants me to

go to Harvard as well. That will…That I will see in the next four years………Yes, but but my main future plan is

to graduate and be a professional programmer. Yes.”

4. Accuracy-oriented Strategies4. Accuracy-oriented Strategies

Utterance of the level 2 student

“……(A pause of about 8 seconds)…I want to teacher…I want much money and big car.

I want……a big house……….(A very long pause of 1 minute and 40 seconds)……….I want to

marriage.”

5. Message Reduction and Alteration Strategies

5. Message Reduction and Alteration Strategies

• Reduce an original message

• Simplify their utterances

• Tend to use familiar words

• Avoid taking the risk of using new or unfamiliar words

5. Message Reduction and Alteration Strategies

5. Message Reduction and Alteration Strategies

Levels Number of Words

Produced

Number of Words

Repeated

5 96 1

4 42 2 (Repeated 4

times and 2 times)

3 40 2 (Repeated 4

Times and 2 times)

2 2 None

5. Message Reduction and Alteration Strategies

5. Message Reduction and Alteration Strategies

Utterances of the level 5 student

“OK, about newspaper. My family will receive…ah..Thai Rat, I think….But I don’t really usually read them because I’m more in the Internet world. I read news from the Internet because they are more updated.

You know up to date, sorry…

5. Message Reduction and Alteration Strategies

5. Message Reduction and Alteration Strategies

Utterances of the level 2 student

“Manager (pronounced with completely Thai accent) newspaper…….(Silence until the time is up.)”

6. Attempts to think in English6. Attempts to think in English

Students of high proficiency:• Can produce a natural flow of speech

• Tend to think in English

Students of low proficiency:• Tend to think in their native language• Construct the English sentences

6. Attempts to think in English6. Attempts to think in English

Levels Number of Words

Produced

Number of Word-for–

Word Translated Sentences

5 76 None

4 22 1

3 39 2

2 13 1

6. Attempts to think in English6. Attempts to think in English

Utterances of the level 5 student

“ Well, basically we watch the TV together…..and we watch…..you know…..after dinner. First we have dinner together as a family and then e go and watch TV together and each of us have our favorite TV show and every night we get to pick

the show we want to see and the rest of the family will watch the show with you and …sometimes we

go out of the city..and that just about it.”

6. Attempts to think in English6. Attempts to think in English

Utterances of the level 2 student

“I watch television with my parent …..(A long pause of 30 seconds)..…….I make….useful…..things with my parent.

Implications on Teaching SpeakingImplications on Teaching Speaking

Goals of Teaching Speaking

“All students should be able to talk accurately and fluently about themselves, express feelings or opinions about things in their daily lives.”

Implications on Teaching SpeakingImplications on Teaching Speaking

Speaking Test = Teaching device

Personal Interview

Role play

Personal Interview Personal Interview

Conducted constantly

Willingness to speak

Fluency

Accuracy

(Myint 1993)

Role Play Role Play

Expose students to conversational English

Encourage students to participate physically

Dramatize a specific situation

Real world of conversational English (Bogart 2006)

ConclusionConclusion

The CU-COA can:

• Assess Thai students’ speaking ability

• Differentiate the students into various levels of speaking ability

• Illustrate different communication strategies