1

Another peek into what today’s English learners as researchers

actually do in production with the scaffolding of 3 e-referencing tools

Hsien-Chin Liou and Chih-Hsin Leehcliu@mx.nthu.edu.tw

CALICO 2006Foreign Languages and Literature,

National Tsing Hua University

2

Overview Purpose of the study

Literature Review

Research Questions

Data Collection

Data Analysis

Conclusion

3

Purpose of the Study Recently the pedagogical effectiveness of referencin

g tools in assisting language learning has attracted quite a lot of attention (e.g., Rundell, 1999; Yang, 2005).

The aim of this study is to investigate the outcome and process of 22 college learners while doing writing and translation tasks. Their preferences and look-up strategies of using 3 web-based referencing tools were documented. The tools were a bilingual dictionary, a bilingual concordancer, and a collocation retrieval program.

4

Literature Review- corpus investigation Chambers (2005) examined the data of the student

s' consultation of the corpora, including choice of search word(s), analytical skills, the problems encountered, and their evaluation of the activity.

Although corpora consultation can complement foreign language learning in various educational contexts, limitations were also found such as the small size of corpora and lack of learner training.

5

Literature Review- Dictionary Look-up Behavior

Dictionary look-ups have been demonstrated as useful scaffolds when learners are working on language tasks (Laufer & Hadar,1997; Rundell, 1999).

Factors of word relevance, word inferability and learners’ vocabulary knowledge were shown to influence look-up behavior (Hulstijn,1993).

Inferring ability is related to word consultation in a less straightforward manner than is learners’ existing vocabulary knowledge.

6

Literature Review- Dictionary Look-up Behavior Rundell (1999) recommended learners to use monolingu

al dictionaries rather than bilingual dictionaries due to their richer information about syntactic behavior, collocation, word frequency, or synonyms.

The bilingualised dictionary yielded the best results on both comprehension and production (Laufer & Hadar, 1997).

When foreign-language readers look up the meaning of unfamiliar words can be influenced by the language task and learner variables.

7

Literature Review- Dictionary Look-up Behavior Frankenerg-Garcia (2005) examined how learners us

ed multiple types of reference materials altogether while doing translation. He used a detailed grid for types of query on various tools and found bilingual types were preferred.

Along the same line of Bland, Noblitt, Armington and Gay (1990), Hulstijn (1993), and Liou (2000a), Liou (2000b) claims that the use of online recording programs to do unobtrusive observation of learner behavior can yield valuable data for applied linguists in understanding what learners actually do on tasks.

8

Literature Review- corpus investigation As technologies advance, online corpus consultation

has become popular and proved useful for language learning.

Previous studies have shown that the concordancer is a useful tool to enhance L2 learners’ vocabulary learning (Cobb, Greaves, & Horst, n.d.; Kennedy & Miceli, 2001; Lee & Liou, 2003) and writing (Gaskell & Cobb, 2004; Kennedy & Miceli, 200; Yeh & Yu, 2004; Yoon & Hirvela, 2004).

9

Literature Review- corpus investigation More research is required to investigate how le

arners behave while working on different tasks with the help of various e-reference tools.

The effectiveness and students’ preferences of using different on-line reference tools should be addressed with mixed research methods so that a more comprehensive picture can be revealed.

10

Research Questions1. Does the online writing practice have an impact on the

students’ performance?

2. What can the computer recording data tell us about the students’ look-ups as a function of tool and task differences?

3. What are the students’ preferences and perceptions of using three online referencing tools in order to complete translation and writing tasks? What are the benefits and difficulties of using them?

4. What does the process data based on the analyses of the selected students’ think aloud protocols tell us?

11

The Study: (1) Participants Twenty-two first-year English majors from an

intact class of a public university participated in this study.

Most of them did not have previous experiences in CALL or corpus learning.

Five voluntary subjects out of the 22 subjects joined two think-aloud sections.

12

(2) Tools and Tasks: 3 Web-based Referencing Tools

  TOTALrecall TANGO Dr.eye

Features A bilingualconcordancer

A collocationretrieval program

A bilingual dictionary

Output Bilingual text Monolingual text Monolingualdefinition with

bilingual examples

Corpus Sinorama (1990-2000)

Sinorama (1990-2000),Voice of America, and

British National Corpus

(see writing practice on Candle for more detailed information http://candle.cs.nthu.edu.tw )

13

Bilingual Concordancer: TOTALrecall

14

Bilingual Collocation aid program: TANGO

15

A commercial bilingual dictionary, popular in Taiwan: Dr. eye

16Figure 1: Illustration of the online writing practice unit

17Figure 2: Illustration of the online translation unit

18Figure 3: Illustration of the online writing unit

19

(3) Data CollectionQuantitative Data Qualitative Data

• the scores of students’ pre-writing and post- writing

• records of the tracker on the frequency of look-up entries

• evaluation questionnaires

• 2 sections of think-aloud protocols of five participants

20

(4) Procedure of the StudyDate Procedures9/14 Pre-writing9/21 Orientation of TANGO & TOTALrecall9/28 Translation (1)10/12 Writing (1)10/14 Think-aloud training10/19 Translation (2), Think-aloud (1), &

Evaluation questionnaire (1)10/26 Writing (2), Think-aloud (2) & Evaluation questionnaire

(2)10/28 Post-writing

21

(5) Data Analyses--Quantitative data analysis (5.1)

Scores of Prewriting and Post-writing

Students’ prewriting and postwriting were graded by two experienced raters using the scale of grading the Test of Written English by the Educational Testing Service with the criteria of 1 to 6.

Table 2. Comparison of pretest and post writing

Mean SD Inter-rater CorrelationPre-test 3.19 0.68 0.83Post-test 4.21 0.61 0.92

22

Table 3. Comparison of writing performance at 2 time points

Sign Test Marked tests are significant at p <.05000

No. of Percent Z p-levelPretest & Posttest 22 95.45455 4.050814 0.000051*

Quantitative Data Analysis (5.2)Scores of Prewriting and Post-

writing Due to the small number of test takers, we us

ed the nonparametric procedure-Sign Test.

23

Quantitative Data Analysis (5.3)

Tracking Data—Language Use

Table 4. The summary of language used for queries for different tasks

Tasks Chinese Both English Error Sum

Writing 622 (55.9%) 27 461

(41.4%) 3 1113

Translation 1109 (66.5%) 30 520

(31.2%) 8 1667

Sum 1731 57 981 11 2780

It seems that the learners used tools more frequently in translation tasks. The number of tool use in translation is about 1.5 times of it in writing tasks.

24

Quantitative Data Analysis (5.4)

Tracking Data- Language Use Users looked up more entries in Chinese for translation, but more English look-ups appeared in writing. This phenomenon may be related to the nature of the tasks.

In translation tasks, users needed to translate the Chinese text into English so that it was necessary for them to use Chinese to search for the English equivalents.

In writing tasks, the learners needed to confirm what they want to express in English so that the frequency of English mode in writing is higher.

25

Quantitative Data Analysis (5.5)Tracking Data- Tool Use

The frequency of use of Dr. eye (the dictionary) and TOTALrecall (bilingual concordancer) were considerably higher in both writing and translation tasks.

Table 5. The use of different tools in different tasks

TypeDictionar

y (Dr. eye)

Collocation aid (Tang

o)

Concordancer (Totalrec

all)

Sum

Writing 50.85% 2.16% 46.99% 100%

Translation 51.53% 4.98% 43.49% 100%

Sum 51.26% 3.85% 44.89% 100%

.

26

Quantitative Data Analysis (5.6)

Tracking Data- Tool Use Seven out of the 22 learners preferred the use of the biling

ual concordancer (TOTALrecall) than the other two tools, while ten of them showed the preference for the dictionary (Dr. eye), and the rest of them used the dictionary (Dr. eye) and TOTALrecall in similar frequency.

Possible factors that influence individual difference in terms of such preference may include learners’ familiarity towards different tools, the design of the tools and the search results, or usefulness of searched information presented by different tools.

27

Quantitative Data Analysis (5.7)

Tracking Data The tracker also recorded students’ use of the on-line referencing tools within one month after the post-writing test.

After the experiment, 12 out of 22 students continued to use these tools.

TOTALrecall TANGO Dr.eye SumTotal 123 38 13 174Percentage 70.69% 21.84% 7.47% 100%

Table 6. Students’ use of the tools one month after the experiment

28

Quantitative Data Analysis (5.8)

Tracking Data

The languages of students used one month after the experiment while querying were roughly in the same frequency.

English Both Chinese Error SumTotal 86 1 83 4 174Percentag

e49.43%

0.57% 47.70% 2.30%

Table 7. The summary of language used for queries one month after the experiment

29

Quantitative Data Analysis (5.9)Evaluation Questionnaire –

Usefulness of Tools

Most participants thought that only one single e-referencing tool might not be enough to assist them and using more than one tool is the most useful strategy.

Table 8. 22 Participants’ perceptions of usefulness of the 3 tools or their combinations

TANGO (1)

TOTALrecall (2)

Dr.eye (3)

1+2 1+3 2+3 1+2+3

Writing 0% 13.63% 22.72% 9.09% 4.55% 36.36% 13.63%Translation 4.55% 4.55% 4.55% 4.55% 18.18% 40.90% 22.72%

30

Quantitative Data Analysis (5.10)Evaluation Questionnaire-

Usefulness of Tools Dr.eye is more helpful in learning the meaning of th

e vocabulary.

TOTALrecall is helpful in learning the usage of vocabulary. TANGO is the second helpful tool, and Dr.eye is the least.

TANGO, for the participants, is a more helpful tool in learning collocation due to its collocation patterns.

31

Quantitative Data Analysis (5.11)Evaluation Questionnaire-

Usefulness of Tools

The participants think that using e-referencing tools is more helpful in improving Chinese-English translation skill than English writing skills.

Dr.eye is the most helpful tool in improvement of translation, TOTALrecall the second.

32

Quantitative Data Analysis (5.12)Evaluation Questionnaire-

Usefulness of Tools Participants think that TANGO is the most helpful tool in

improvement of English writing skill (which contradicts to what they would prefer to use in the future).

The participants reported that they did not know how to use TANGO because they were not very familiar with the functions of the tool.

More intensive orientation and practice would be the first priority for students to familiarize the functions of the tool.

33

Quantitative Data Analysis (5.12)Evaluation Questionnaire

For future use, students generally hold positive attitudes towards using these e-referencing tools in the future.

Mostly they would like to have more than one e-referencing tools, especially when doing translation tasks.

34

Qualitative Data Analysis (1)Think-aloud Protocols

A coding scheme adopted from Frankenberg-Garcia (2005) and modified was taken up in order to identify and classify their look-up strategies.

The coding scheme includes seven categories: 1. Finding an L2 equivalent; 2. Confirming a hunch; 3. Finding a suitable collocate; 4. Choosing the best alternative or synonym; 5. Checking spelling; 6. Retrieving linguistic info from examples; 7. Looking for part of speech.

35

Qualitative Data Analysis (2)think-aloud protocols

Each individual concept was identified, in terms of each individual problem to be solved in writing or translation no matter how many entries they entered.

Regardless of task difference in writing or translation, students’ look-up behavior falls into three major categories: finding an L2 equivalent (50%), confirming a hunch (19%) and retrieving linguistic information from examples (11%).

36

Qualitative Data Analysis (3)think-aloud protocols

In order to obtain information students needed, they tended to do cross referencing very often, which is also reflected in our tracker data.

Each e-referencing tool provides different types of linguistic information. The information each e-referencing tool provided might not be sufficient. Cross-referencing could help students to test and confirm their hypothesis while searching.

37

Qualitative Data Analysis (4)Think-aloud Protocols

Successful use of strategy was defined as finding the correct information and applying it in writing or translation correctly.

The results show that 89% of these 5 students’ problems were solved and applied successfully.

38

Qualitative Data Analysis (5)think-aloud protocols

Among 89% of successful use of strategy, 3 categories: finding an L2 equivalent (43%), confirming a hunch (18%), and checking spelling (9%) were found to be more often used.

Other strategies that were not used successfully might need to be trained.

39

Triangulation of All the Data (1)

The participants also reported that when having problems in English writing, they would tend to search Dr.eye first, then TOTALreacll.

And Dr.eye and TOTALrecall could usually provide the information they needed.

The data confirmed that they would do initial look-up of perhaps an English equivalent of a Chinese entry first, followed by further consultation of how the word should be used in context using either TOTALrecall or TANGO.

40

Triangulation of All the Data (2)

We observed several different types of problems students had while searching for information: unfamiliarity with tools themselves, or with searching techniques, the small size of corpora in the tools, over-reliance on the tools, and carelessness in searching.

The questionnaire responses also pointed to some difficulties that students encountered while using these e-referencing tools.

Only 22.72% of students perceived no difficulty in using the tools, whereas 36.36% of students felt that they had difficulty in using TANGO and 18.18% of students in using TOTALrecall.

41

Conclusion The statistical comparison indicated this group

made significant improvement (the learning products) perhaps through the design and the use of the online writing practice tasks.

It seems that online writing practice with scaffolds of referencing tools can improve their production as evidenced in the learning product data.

42

Conclusion

The acceptance of the innovative corpus tools for references is satisfactory to the learners based on the questionnaire data; definitely more orientation with practice from English teachers is needed for students to acquire mature look-up skills.

Although succinct task differences between writing and translation were not observed, they warrant more future research.

43

Acknowledgements The paper was funded by a National Science Council

project (under the number of NSC 94-2524-S007-001, the CANDLE project).

We also thank Hsin-Ping Yu for programming and sorting the tracked data, Jany Lin for helping collecting and analyzing the data, and Ashley Ho for serving as another rater for grading the writing and coding the think-aloud protocols.

The 22 participating students are also acknowledged.

44

Thank You for Your Attention!