Evaluating the Integration of Technology and Second Language

Volume 17 Number 2 269

Adair-Hauck, Willingham-McLain, and Youngs

Evaluating the Integration ofTechnology and Second

Language Learning

Bonnie Adair-Hauck, Ph.D.University of Pittsburgh

Laurel Willingham-McLain, Ph.D.Duquesne University

Bonnie Earnest Youngs, Ph.D.Carnegie Mellon University

ABSTRACT

This article reports the findings of a program evaluation project that as-sessed the integration of technology-enhanced language learning (TELL)into a second semester, college-level French course. Thirty-three French IIstudents participated in this study. Students in the treatment group metwith the instructor three days per week and, for the fourth class, theyparticipated in TELL activities. The control group met with the instructorfour times per week. Both groups had the same instructor, textbook, andancillary materials. The article reports on student performance in Frenchfor listening, speaking, reading, and writing skills, and on their culturalknowledge. Findings indicate that the students in the treatment groupperformed equally well as the control group in listening and speaking andbetter on reading and writing achievement measures. The study also in-cludes findings regarding student motivation, anxiety, and perceptions onmeeting the language learning goals students set for themselves. The re-sults may be interpreted that it is both feasible and desirable to integrate,in principled ways, TELL activities into the language learning curriculum.

KEYWORDS

Technology-Enhanced Language Learning, Multimedia Curriculum, Inte-gration of Technology, Program Evaluation, College-Level French Teach-ing, Computer-Assisted Language Learning, Computer-Mediated Writing

© 1999 CALICO Journal

270 CALICO Journal

Evaluating the Integration of Technology

INTRODUCTION

Background

Although the use of personal computers has proliferated into the work-place as well as in academe, there is a paucity of published research oncomputer-assisted language learning (CALL). Conrad (1996) reviewedseveral professional journals (Modern Language Journal; Foreign LanguageAnnals; Unterrichtproxis; French Review; Hispania; and System) publishedfrom 1992-1995 and reports that only 1.4% of the articles published ad-dressed empirical CALL studies. The majority of CALL articles are prac-tical in nature and showcase a software program or suggest guidelines forthe implementation of software. The few empirical CALL studies pub-lished have focused on topics such as feedback (Brandl, 1995; Nagata,1993; Nagata & Swisher, 1995; Robinson, 1989, 1991); pronunciationand intonation training (Stenson et al., 1992); interactive reading(Svenconis & Kerst, 1995); computer-facilitated student interaction andtalk (Beauvois, 1992; Chun, 1994; Cononelos and Olivia, 1993; Meunier,1994); writing (Armstrong & Yetter-Vassot, 1994; Barnett, 1989; Greenia,1992); and cognitive style (Liu & Reed, 1994; Raschio, 1990).

Previous studies on technology and language learning have comparedthe effectiveness of a new technology with more traditional modes of learn-ing. In general, these studies have examined learning over a short timeperiod such as a few days. For example, Raschio (1990) explored the cog-nitive style (field dependent or field independent) of 62 first semesterSpanish students. The goal of both the control and experimental groupswas to learn how to form Spanish direct and indirect object pronounsover a two day period; day three was reserved for testing. The results didnot reveal any statistically significant relationships between the level offield dependence, mode of instruction (printed vs. CALL), and studentachievement. Although the study did unveil some interesting findings re-garding student attitudes, Raschio suggests that the profession needs tomodify its traditional research framework to include process variables,not just outcomes in second language learning.

Besides a lack of research-oriented CALL articles, few published evalu-ations of second language (L2) programs have integrated CALL into thecurriculum. Johnson (1985) conducted a study on computer-assisted learn-ing to promote L2 acquisition, but her report focused primarily on suchissues as equal access, software development, computers in composition,typical practices, and model programs. There appears to be no publishedarticle on language program evaluation intended to assess language skilldevelopment and the integration of technology into the curriculum.1

At the beginning of this decade, Garrett (1991) articulated a number ofresearch questions regarding program development and technology, “What



kind of software, integrated how and into what kind of syllabus, at whatlevel of language learning, for what kind of language learners, is likely tobe effective for what specific purposes?” Herron and Moos (1993) reiter-ated the aforementioned concerns by stating that “One of the major con-cerns facing the foreign language teaching and literature profession is howto integrate new technological advances into instruction.”

This article reports the findings of a longitudinal program evaluationproject that focused on L2 skill development (listening, speaking, read-ing, writing, and culture) and the integration of technology-enhanced lan-guage learning (TELL). These findings can serve to inform foreign lan-guage departments as they make important decisions concerning the useof technology in their programs.

Innovations in Technology Application

In the 1980s, CALL software featured a behavioral or stimulus-responseapproach to language learning by stressing mastery of grammatical prop-erties or discrete-point learning (Conrad, 1996; Johnson, 1992; Pusack &Otto, 1997). Much of the software was grounded in a transmission ap-proach to learning. For example, Johnson (1992) reviewed the ESL soft-ware for language minority children and found that in general “these pro-grams focused on grammatical forms, many of which were not that par-ticularly useful.” Recently, tremendous progress has been made in tech-nology, and the profession has broadened its interest in the role of tech-nology to encompass a multimedia approach by including computers withCD-ROM drives, videodisc players, video players, and hypermedia capa-bilities. In this respect, the preferable use of multimedia has evolved froman “add on” component (CALL) to a fully integrated feature of the foreignlanguage curriculum (TELL).

Pusack and Otto (1997) advocate that “the strength of multimedia isthe synergy derived from presenting content using a variety of modalities(listening, reading, writing, and speaking) that are linked together in mean-ingful ways [italics added] to provide an in-depth experience.” Multime-dia also has the advantage of providing students with longer stretches oforal and written discourse embedded within a rich visual cultural contextfor communication (Garrett, 1991). Technological advances, such as theWorld Wide Web, connect our students to up-to-date authentic realia andto native speakers. Multimedia thus has the capability to stretch our cur-riculum beyond the traditional walls of the classroom and to integratemuch needed sociolinguistic authenticity into our programs (Meunier,1994).

Besides expanding its orbit of interest to include a multimedia approach,the profession has also redefined the role of technology in L2 learning.

272 CALICO Journal


Similar to the reoriented role of the teacher from central to supportive inthe language learning process, we have witnessed a metamorphosis in therole of the computer from “transmitter of knowledge and skills” to a toolthat supports and assists learners to complete tasks (DeVillar & Faltis,1991). The profession now realizes that a computer is a medium for learn-ing and not a method for L2 instruction.

Although there are numerous benefits of integrating technology into thecurriculum, we will address three specific areas within the context of ourprogram evaluation: (a) computer-mediated writing, (b) the integrationof culture, and (c) the sociocultural benefits of technology.

Computer-Mediated Writing

The research concerning computer-assisted composing and the devel-opment of L2 writing has produced diverse results. While some studiesindicate that students spend more time and write more at the computer(Hawisher, 1989; Roblyer et al., 1988), some writers tend to plan less andgive less attention to their ideas or content (Haas, 1989; Theismeyer, 1989).Costanzo (1994) stresses that newer word-processing software incorpo-rates a process-oriented approach to writing as a recursive process of dis-covery, elaboration, and revision. Bernhardt et al. (1989), Phinney (1988)and Rodrigues (1985) have stressed that many of the studies that reporton computer-mediated writing have been short-term studies. Phinney(1991) argues that at least one semester is needed for positive improve-ment in students’ writing to occur. Phinney (1989) also notes the impor-tant role of the teacher to serve as a guide in assisting learners on how touse computers in the writing process, “Without specific instruction in us-ing the computer to facilitate the writing process, from prewriting to revi-sion, the computer alone, appears to have little effect in changing thewriting behavior of naive writers.”

Studies of student motivation and attitude toward writing have pro-duced more positive results (Daiute 1984; Neu & Scarcella; 1991). Forexample, Phinney (1991) found that computer composing reduced block-ing problems and improved the students’ overall attitude toward writingin English. Phinney noted, however, the importance of the teacher’s role inproviding writing activities during regular class time which stressed brain-storming, drafting exercises, peer-commenting, and revising of drafts. Stud-ies investigating computer networks used in L1 and L2 classes underscorethat these networks encourage more written discussion and therefore moreinvolvement on the part of the learner (Bump, 1990; DiMatteo, 1990;Kelm, 1992).



Integrating Culture through Technology

Multimedia capabilities such as audio/videotapes, CD-ROMs, videodiscs,the World Wide Web, E-mail exchanges, and electronic conferencing, en-able the profession to incorporate much needed sociolinguistic authentic-ity into the L2 classroom (Meunier, 1994). Lafford and Lafford (1997)explain that access to multimedia capabilities “facilitates the learner’s un-derstanding of the various social and psychological forces at work todayin the target culture and provides a context in which students can inter-pret the behavior of the target culture inhabitants.” These authentic mate-rials can serve as the tools to frame meaningful, communicative class-room-based discussions. Furthermore, the use of video exposes studentsto native speakers using appropriate interactional communication strate-gies including nonverbal or kinesthetic behaviors.

Language Learning as a Social Phenomenon

Since language is a social phenomenon, language learning occurs throughsocial interaction involving teachers and more capable peers (Tharp &Gallimore, 1988; Vygotsky, 1978; Wertsch, 1979). Similarly, multimediais now seen as a “tool” to enhance communication and discovery-orientedlearning, much of which can be accomplished by working cooperatively insmall groups. In this way, multimedia is a tool which assists the learners asthey work in their zones of proximal development (Vygotsky, 1978). Co-operation with a teacher, more capable peers, or with multimedia allowsstudents to perform tasks that they would be incapable of performing ontheir own. Generally, these shared activities need to be followed by indi-vidual practice on a similar task until learners can perform the task inde-pendently (Cummins, 1991). Multimedia can thus assist learners to movefrom the interpsychological to the intrapsychological level (Vygotsky, 1978).

Johnson (1991) found that integrating computers into the classroomcan promote cooperative learning.

Rather than isolating and promoting asocial behavior, as manyhad feared, there is a growing body of evidence that computer usecan promote new ways of working together, productive peer teach-ing, as well as high quality social and academic task-based inter-action, and that these kinds of interaction are related to higherlevels of interest, motivation and achievement.

The teacher, of course, plays a critical role in selecting computer softwarethat will influence the sociocultural dynamic of cooperative group work(Johnson, 1992; Mydlarski, 1987; Piper, 1986).

274 CALICO Journal


Foreign Language Program Evaluation

Brown (1989, 1995) defines language program evaluation “as the sys-tematic collection and analysis of information necessary to improve a cur-riculum, assess its effectiveness and efficiency, and determine participants’attitudes within the context of a particular institution.” Brown (1989)continues by saying that “formative evaluation occurs during the develop-ment of a program; the purpose of gathering information is to improve theprogram. Summative evaluation occurs at the end of the program, and thepurpose of gathering information is to determine whether the programwas successful.”

Various evaluation models exist as well. A participatory model of evalu-ation can be preferable to an outside expert model because the latter oftenencourages adversarial roles between the expert and the teacher. Conse-quently, Alderson and Scott (1992) recommend a collaborative or partici-patory model which includes a consultant who provides expertise andguidance along with the teacher(s) of the program. A participatory evalu-ation model “centers on insiders though benefitting from the advice ofoutsiders.” Ross (1992) concurs with the notion of a collaborative or par-ticipatory model for “it decreases the teachers’ anxiety of being ‘watched’and allows direct participation by the teacher as both practitioner andobserver.”

In addition, Beretta (1986) summarizes the strengths of two frameworks,field research and laboratory research, and argues for more field researchin language program evaluation. Beretta notes that field research is “long-term, classroom-based inquiry into the effects of a program” and that labo-ratory research is “short-term and only involves the testing of individualcomponents of a theory in an environment in which extraneous variablesare artificially held constant.”

A further consideration is the timing of the evaluation: during or afterthe program or both and for how long? Brown (1995) examined languageevaluation programs between 1988 and 1994. Thirteen of the 16 languageevaluations were longitudinal in nature and ranged from two months tothree and a half years. These evaluations were conducted during the pro-grams. Evaluations can also occur at the end of a program. For example,Snow and Brinton’s (1988) evaluation used a retrospective format afterstudents had completed a content-based program. Pitiyanuwat (1986) notesthat the profession needs to gather data from students after they havefinished a program and actually have to use the language. Brown (1995)suggests that the best framework would include evaluation during the pro-gram, immediately after, and later in a follow-up phase.

With regard to data collection, Alderson and Scott (1992) point outthat quantitative data are “both easier to gather, and more amenable toanalysis and summary.” However, a number of researchers (Beretta, 1986;



Donato, Antonek, & Tucker, 1996; Lynch, 1990; Ross, 1992) stress theneed to provide multiple perspectives by triangulating data. As Ross (1992)argues, “both qualitative and quantitative types of evaluation can be uti-lized with a view to defining a program,” and “both may provide alterna-tive views of the same classroom phenomena.”

Finally, Brown (1995) defines the evaluation of product as “any evalua-tion which is focused on whether the goals (product) of the program wereachieved, as contrasted to process evaluation which centers on what isgoing on in the program (process) that helps to arrive at those goals (prod-uct).” Long (1984) argues for doing both when he states “using processand product evaluations in combination, one can then determine not onlywhether a program really works, or works better, but if so, why, and if not,why not?”

THE PROJECT

Project Rationale and History

We conducted a semester-long evaluation of the integration of TELLinto a second semester French class. We chose to examine the impact ofreplacing one class per week with multimedia activities for four reasons:(a) the paucity of empirical research on the effects of TELL over time, (b)the mandate from our university to make learning remote in time andspace, (c) the difficulty students had enrolling in four-day-a-week courses,and (d) the success we had had with computerized grammar and vocabu-lary exercises in our self-paced courses. Informal observation indicatedthat even though the self-paced students had minimal contact with a pro-fessor and other students, many of them excelled in their writing. Somestudents mentioned using Dasher2 exercises as a way to prepare for writ-ten portions of the course exams.

Endeavoring to be thorough in our program evaluation, we drew on thestrengths of multiple evaluative procedures. Although our evaluation wasprimarily summative in that we sought to establish whether the TELLcurriculum was successful in meeting our goals, we also conducted for-mative evaluations in order to improve continually the TELL components.According to a participatory model, we did field research in a classroomcontext throughout the entire semester. We collected both quantitativeand qualitative data in order to assess both the process and product oflanguage learning.

276 CALICO Journal


THE PILOT STUDY

During our spring 1996 pilot study, students in one section of second-semester French at Carnegie Mellon University (CMU) participated in thefollowing multimedia activities in lieu of the fourth contact hour: comput-erized grammar and vocabulary exercises using Dasher, instructional vid-eos, and optional use of an on-line spell-checker and French-English glos-sary. A control group followed the traditional syllabus, meeting four timesper week. Qualitative and quantitative data on student performance, anxi-ety, and motivation were gathered to determine whether segments of lan-guage courses could be reasonably expected to become “remote in spaceand time” and still ensure that students develop an appropriate level ofproficiency in all skill areas.

We found parity on the measures of anxiety, motivation, listening, read-ing, and writing. Although both groups showed significant improvementon the posttest of cultural understanding, the treatment group showedsignificantly more improvement than the control group (p = .019). Fur-thermore, there was a strong trend toward significance (p = .066) of thetreatment group scoring better on the semester-end speaking test. Giventhese encouraging results, we revised the course and the study design toanswer more research questions. Specifically, whereas Dasher exerciseshad been the primary focus of the pilot study, the study currently beingreported incorporated a rotation of multimedia-based reading, grammar,vocabulary, and listening exercises with systematic speaking and writingfollow-up activities.

Research Questions

The primary purpose of our study was to investigate the feasibility anddesirability of replacing the fourth contact hour with multimedia activi-ties outside the classroom. Our evaluation of this program was intendedto answer the following questions:

1. Would the TELL experimental group students perform as well asthe control group in listening, speaking, reading, writing, and cul-tural understanding?

2. How would TELL affect student foreign language motivation andanxiety?

3. How would TELL affect students’ perception of meeting the lan-guage learning goals they set?

4. In what ways would TELL modify the roles of the teacher andstudents?

5. More generally, would the TELL course be an effective curriculardesign for achieving second language development in this context?



Design of the Study

DESCRIPTION OF STUDENTS

We began the study by administering a demographic and language back-ground questionnaire in order to determine whether the treatment group(n = 17) and control group (n = 16) were similar. There were no signifi-cant differences between the groups in terms of sex, age, grade point aver-age and year in college. There were 5 women and 12 men in the treatmentgroup, and 7 women and 9 men in the control group. The students rangedin age from 19 to 33, but most were of traditional college age. Studentshad declared majors in five of the seven colleges at CMU, representing thehumanities, social sciences, natural sciences, technology, fine arts, andbusiness. Despite CMU’s reputation for technology, 21 of these 33 stu-dents were majoring in the humanities, social sciences, or fine arts. Everystudent was familiar with computers; 21 owned a computer, and only onestudent in the treatment group reported not being comfortable using acomputer.

Although the majority of the students in both groups were American bybirth and native speakers of English, many students reported knowingother languages. Out of the total number of students, only two from thecontrol group reported that most of their experience in French had beenoutside the classroom. Half of the students in each group had taken thefirst semester French course with the researcher/instructor the previoussemester. This facilitated conversation among the students, since manyalready knew each other and were familiar with the teaching style andgeneral course requirements.

COURSE DESIGN

We conducted the research over the course of the spring 1997 semesterin a second semester French course, which used chapters 6-10 of Allons-y (Bragger & Rice, 1996). The treatment group met three days a weekfrom 10:30 to 11:20, and, in lieu of the fourth class period, students chosewhen and where to do the multimedia components. The control groupmet four days a week, from 12:30 to 1:20, and prepared the same multi-media components that were assigned to the treatment group, but in class,on paper, or by watching the video as a group. Both groups were assignedidentical writing homework and follow-up speaking tasks. The controlgroup completed these tasks in class and for homework. The treatmentgroup prepared the assignments on their own time. For both groups, thespeaking tasks were evaluated in class, and the writing tasks were col-lected and graded.

The challenge was to incorporate systematically the four language skillsand cultural understanding into the students’ work on a weekly basis. Wedeveloped a 12-week rotation of three TELL components that were famil-

278 CALICO Journal


iar and had been successful in past courses: the CMU French Reader,Dasher exercises, and the Allons-y video.3 In addition, we had alreadytested the materials sufficiently to know that after a brief orientation, stu-dents were unlikely to encounter technical difficulties. We also chose thesecomponents because they are accessible to many instructors interested inimplementing them in a traditional classroom setting.

The CMU French Reader, a computerized reading program featuringauthentic journalistic texts, pre- and postreading questions, a glossary(French-English, English-French), culture notes, grammar notes, and awriting pad was created by Bonnie Youngs, the course instructor, andChristopher M. Jones (1996). This HyperCard-based program is housedon the Modern Languages server and is accessible on any Macintosh com-puter on campus. We used the Reader to introduce each Allons-y chapterthrough a text relevant to the chapter theme (e.g., food, lodging).

The second multimedia component in the cycle consisted of Dasherexercises developed by CMU faculty. Dasher (Pusack & Otto, 1992) is anauthoring program which permits an instructor to write exercises of vari-ous types (e.g., multiple-choice, fill in the blank, sentence completion,listening comprehension, written and visual-based text). For this project,we assigned Dasher exercises that emphasized the grammar and vocabu-lary of each chapter.

The third TELL component was the Allons-y videocassette (Heinle &Heinle, 1996). The video presents brief scenarios for each chapter per-formed by native speakers in France and Guadeloupe speaking at naturalspeeds. Students had previewing questions and listening comprehensionexercises to do while watching the video. We offered the video last in thecycle because it provided a synthesis of the chapter content.

For each of the 12 weeks, students in the treatment group received a labassignment sheet outlining (a) their reading, grammar/vocabulary, or lis-tening comprehension task and (b) their weekly writing homework andfollow-up speaking task (to be prepared for the next class period). (Seethe Lab Assignment Sheet in Appendix A.) The writing and speaking ac-tivities provided a way for students to apply and develop what they hadlearned through their own personal expression in simulated authentic tasks.

The lab assignment sheet also contained a checklist of the tools studentsused while doing the lab, for example, their textbook, paper dictionary,electronic glossary, and spellchecker. The English-French/French-Englishelectronic glossary (Carnegie Mellon University, 1996) is integrated intothe CMU French Reader, but also can be accessed as a stand-alone pro-gram. The Spellchecker is a foreign language spell checking program. Inaddition, for each week’s lab, we asked students to indicate the amount oftime they had spent, to rate the lab on a scale of 1 “poor” to 5 “great,” toexplain their rating, and to describe in a few words how they felt whiledoing the assignment.



RESEARCH DESIGN

In order to test our primary hypothesis that students using the TELL com-ponents in lieu of one class period per week would perform French tasksin all the skills as well as students meeting four times a week, we collecteda variety of qualitative and quantitative data early, throughout, and nearthe end of the semester (see Table 1).

Table 1Overview of Data Collected from Treatment and Control Groups

Early-semester During-semester End-of-semesterdata data data

Foreign Language 24-item measure 24-item measureMotivationa

Foreign Language 20-item measure 20-item measureAnxietyb

Goal Setting/Attainment & open-ended open-endedAffective Questionnairec questionnaire questionnaireCultural Knowledge Quiz QuizListening 5 achievement testsSpeaking Paired Test Paired TestReading 5 achievement testsWriting Writing on first lab 5 achievement tests Writing on last

and on first weekly lab follow- lab and lastachievement testd up activities achievement test

Time on Task 12 weekly labs(treatment grouponly)

Students’ Perspectives 12 weekly labs Open-ended(treatment group questionnairee

only) Follow-upinterviews (se-lected students,treatment grouponly)

aThe Foreign Language Motivation measure was adapted from Tremblay and Gardner (1995;personal communication, November 1995).bThe Foreign Language Classroom Anxiety Scale was adapted from Horwitz and Young(1991).cThe Goal Setting and Affective Questionnaire was adapted from Chamot (personal com-munication, November 1995). We also collected information about learning styles usingReid’s Perceptual Learning Style Preferences Questionnaire (1987), but these data werenot conclusive.dBoth groups had the same weekly homework writing assignments and test items. An inde-pendent rater used an adapted form of the ESL Composition Profile (Glisan, 1981) to rateearly and late semester writing samples. Note that the first and last writing test items arealso reported as part of the MANOVA analysis of the achievement tests from throughoutthe semester.eStudents’ perceptions of improvement in listening, speaking, reading, writing, culturalknowledge and self-confidence were gathered on an end-of-semester open-ended question-naire. (See end-of-semester questionnaire in Appendix D.)

280 CALICO Journal


As noted earlier, we first established that there were no significant de-mographic differences between the two groups and could thus comparethe language performance of the two groups using t-tests and MANOVAwithout having to adjust for initial differences between them. Recogniz-ing that the instructor and students were there first and foremost for do-ing a second semester French course, and only secondarily for carryingout our study, we balanced the number of data collection instruments withthe need for time-on-task learning French.

Instead of viewing the classroom context as a contaminated researchenvironment, we chose to describe, in the ethnographic tradition, as manyof the variables as possible so that readers could themselves interpret ourdata and determine the relevance (or “generalizability”) to their own in-structional contexts. In order to support each finding with evidence fromdifferent sources, we collected scores on pre- and posttests, grades onclassroom chapter tests, self-reports from the students and instructor, in-terview data from a sample of students, student evaluations of each labassignment, check-lists of materials used, and records of time-on-task.

Our research team consisted of the instructor/researcher who was largelyresponsible for developing the curriculum and the TELL components andtwo researchers who designed the study and conducted most of the dataanalysis. Two student researchers observed the classes, summarized muchof the qualitative data, and coded the quantitative data. An independentrater analyzed the writing samples.

Findings

EVALUATING LEARNER MOTIVATION AND ANXIETY

In addition to demographic background, we examined whether thegroups differed on measures of foreign language motivation and anxiety.Both measures consisted of statements answered using a Likert scale rang-ing from 1 “strongly agree” to 5 “strongly disagree.” We adapted 18 itemsabout students’ motivation to learn French from Tremblay and Gardner’squestionnaire designed for students of French in Canada (1995; personalcommunication, November 1995). These items reflected various aspectsof their expanded construct of language learning motivation (e.g., desireto learn French, motivational intensity, persistence, and instrumental andintegrative orientations). We then added six technology related questionssuch as “The thought of using various types of technology to learn Frenchsounds interesting,” and “I would rather spend my time on activities thatdo not require the use of technology,” to determine whether there weredifferences between the two groups in their motivation to use technologyas a means of learning French.



To examine language-learning anxiety we chose items from Horwitz andYoung’s (1991) “Foreign Language Classroom Anxiety Scale” and onceagain added questions about technology and language learning such as“Doing French exercises on the computer sounds like fun,” and “Thethought of using various types of educational technology to learn Frenchscares me.” There proved to be no significant difference in motivation oranxiety between the groups either on the entire set of items, or on thetechnology clusters alone, at the beginning or end of the semester. Fur-thermore, there was no significant change over the semester in motivationor anxiety.

On the final questionnaire, one student explained how TELL helped tolower her level of anxiety, “Last semester, when we did the video as awhole class, I didn’t have a clue what was going on. But this semester, Icould do it by myself, or with a friend, and it became so much easier. Now,I’m not afraid of the listening sections for our tests. I used to be so scaredof those.”

EVALUATING LEARNERS’ ATTAINMENT OF GOALS

Very much linked to students’ motivation to learn French and their lan-guage learning performance is the setting of specific goals (Tremblay &Gardner, 1995).4 Therefore, using a simple open-ended questionnaire, weasked students to state their personal goals for this course: “Why are youtaking elementary French II? Please state your projected goals for thiscourse. Be as specific as possible.” Near the end of the semester, we asked,“This is the end of the semester in elementary French II. Do you believethat you have made progress toward your projected goals for the course?”

The goals students reported were similar for both groups. The majority(treatment group, 10 of 17 and control, 11 of 16) had linguistic goalssuch as gaining fluency, especially in spoken French. Only one or two ineach group stated that they were taking French because it fulfilled a gen-eral education requirement. (There is no foreign language requirement atCMU.) Other goals included learning about French/Francophone culture,traveling, using French in their careers, and living abroad.

At the end of the semester, 14 of the 17 students in the treatment groupreported that they had met their goals, two said they almost had, and onesaid he or she had not met the projected goals. In the control group, 11 ofthe 15 students who filled out the final goals questionnaire stated theyhad made clear progress, and four students had made some progress, butnot as much as they had hoped for. Treatment group students reported, “Iam more confident and not afraid to speak,” “I’m on my way to beingfluent,” “I have sharpened my French skills,” “I am more confident in myreading, writing, speaking skills,” “I have almost met my goals.” Control

282 CALICO Journal


group students reported, “Yes, because my vocabulary has increased,” “I’mon my way to being fluent,” “I built a foundation for learning the lan-guage,” “I have made progress but would have liked to have made more,”“Not as much as I would have liked.” More students in the treatmentgroup were positive about having made satisfactory progress toward theirprojected goals.

On the final questionnaire, students reported that having the flexibilityto individualize the use of the multimedia activities played an importantrole in their progress and development of certain language skills: “One ofthe great things about the multimedia activities is that you can spend asmuch time on them as you want,” and “I liked the flexibility of the multi-media activities, so that if you feel that you need more practice, you cando it again and again.” The multimedia activities challenged students totake greater responsibility in working toward their goals for learning French.

EVALUATING KNOWLEDGE OF CULTURE

We designed a culture quiz to diagnose knowledge of both general factsabout the Francophone world, and specific facts about topics treated thatsemester (e.g., university studies, shopping, lodging). (See the culture quizin Appendix B.) The treatment group significantly outperformed the con-trol group on both the prequiz and postquiz, with average scores of 68.4%and 76.0% versus 57.0% and 61.9% respectively (see Table 2).



Table 2Cultural Knowledge, Speaking, and Writinga

Early-semester data End-of-semester dataMean SD Mean SD

Cultural KnowledgeTreatment (n = 17) 68.4 13.5 76.0 12.9Control (n = 16) 57.0 11.0 61.9 13.3

p < .05 p < .005b

Speaking TestTreatment (n = 17) 80.8 7.8 86.1 6.2Control (n = 16) 78.9 13.8 91.1 7.2

n.s. p < .05b

Test WritingTreatment (n = 17) 17.41 2.06 18.94 2.03Control (n = 16) 17.31 2.36 18.38 2.09

n.s. n.s.

Homework WritingTreatment (n = 14) 15.21 3.29 17.93 2.30Control (n = 12c) 18.25 2.74 16.92 4.10

p < .05 n.s.d

aWe used t-tests to examine differences in mean scores between the treatment and controlgroups.bWhen change over time was taken into consideration, there was no significant differencebetween the treatment and control groups in cultural understanding and speaking.cWhereas all the students took the exams, several students did not turn in their homework.dThere was a significant difference between the groups in the way they changed over time;the control group’s homework writing scores decreased, and the treatment group’s writingscores increased.

However, there was no significant difference between the two groups inaverage gain made over the semester (computed as postquiz score minusprequiz score).

EVALUATING SPEAKING SKILLS

Early and late in the semester, students took an audio-recorded, pairedspeaking test administered by the instructor and then rated blindly byanother member of our research team. The speaking test focused on func-tional use of language and meaning-making and consisted of three parts:picture description, a role play, and past narration cued by a series ofpictures. (See the oral test in Appendix C.) The early and late tests weresimilar in format, but the tasks were somewhat altered such that students

284 CALICO Journal


had to use the same skills in new contexts. On the pretest, there were nosignificant differences between the average scores of the treatment andcontrol groups (80.8% vs. 78.9%). Although the control group performedsignificantly higher on the posttest (treatment group 86.1% and controlgroup 91.1%), there was no significant difference between the groups ingain made over the semester, computed as posttest score minus pretestscore (see Table 2).

EVALUATING LISTENING, READING, AND WRITING

Throughout the semester, students took five achievement tests, eachone testing listening, reading and writing, all at a paragraph length dis-course level. These tests were designed and graded by the instructor.5 Weused a Multivariate Analysis of Variance (MANOVA) to examine the dif-ference between the groups’ performance on these tests in each skill. Inlistening, there was no significant difference between the two groups (seeTable 3).



Table 3Listening, Reading, and Writing Testsa

Listening Reading Writingb

Mean SD Mean SD Mean SD Test I

Treatment 83.7 6.9 91.7 7.7 80.5 7.8Control 74.3 15.0 78.4 11.1 88.6 5.1

Test IITreatment 90.2 18.5 91.7 5.9 80.0 6.7Control 90.6 12.9 78.4 8.1 88.4 5.8

Test IIITreatment 59.6 11.3 99.3 3.0 79.9 8.0Control 65.0 11.5 98.3 4.4 77.3 12.1

Test IVTreatment 88.5 17.5 99.8 1.0 73.5 10.0Control 75.3 28.4 99.7 1.0 62.3 10.0

Test VTreatment 80.9 16.1 84.6 16.2 82.6 7.3Control 72.4 17.3 86.2 15.4 79.4 8.3

Note: For all tests, Treatment Group (n = 17) and Control Group (n = 16).

aWe used MANOVA to examine differences between mean scores for the treatment andcontrol groups in listening, reading and writing. For listening, there was no significantdifference between the groups. The treatment group scored significantly better than thecontrol group on reading (Wilks’ Lambda = .41; F(5,26) = 7.62, p < .001) and writing(Wilks’ Lambda = .24; F(5,26) = 16.18, p < .001).bThe writing on the first and last test was also analyzed by an independent rater using adifferent scale. See Table 2.

In reading, the treatment group performed significantly better than thecontrol group (Wilks’ Lambda = .41; F(5,26) = 7.62, p. < .001). Similarly,in writing, the treatment group outperformed the control group (Wilks’Lambda = .24; F(5,26) = 16.18, p. < .001).

In order to further investigate the difference between the groups in writ-ing, we had an independent rater analyze the writing on the first and lastachievement tests and the first and last homework writing assignment(follow-ups to labs 1 and 12 for the treatment group and the analogoushomework done by the control group).6 There was no significant differ-ence between the group means on the first test, on the final test, nor in theamount of improvement shown by each group over time.

In the writing homework, however, the control group’s mean on thefirst assignment was significantly higher than that of the treatment group(treatment group mean = 15.21; control group mean = 18.25, p < .05).

286 CALICO Journal


The pattern of change from the first to the final homework assignmentdiffered significantly for the two groups (p < .05). The treatment groupscored higher on the final assignment than on the first, and the controlgroup’s scores decreased (experimental group mean = 17.93; control groupmean = 16.92). Given that both groups performed equally well on thetests, clearly one group did not learn to write better than the other. In-stead, it appears that at the end of the semester, treatment group studentswere more motivated than control group students to do quality writing ontheir homework, even though it counted for less of a grade than the tests.Indeed, the instructor had noted that the treatment group wrote longerand more complex homework compositions than the control group.

TIME ON TASK DOING THE TELL COMPONENTS

For each of the twelve weeks of TELL labs, the treatment group filledout a lab assignment sheet. (See the Lab Assignment Sheet in AppendixA.) Table 4 portrays the systematic cycling of the Reader, Dasher exer-cises and Allons-y video for each of the five chapters.



288 CALICO Journal


In three instances, two of these components were combined into oneweek in order to fit the 15 labs into 12 weeks.

Because the treatment group students were working outside of class, weasked them to report the amount of time they spent on the lab assignmentso that we could compare their time on task with that of the control group,who used the majority of 50 minute class periods doing the in-class activi-ties described earlier.7 With the exception of the second chapter (Chapter7), on which the average amount of time spent was 44 minutes, the aver-age amount of time spent on each of the three tasks per chapter did notvary more than 7 minutes from the overall mean of 34.6 minutes. Thecontrol group spent about 40 minutes per week on the in-class activities,analogous to what the treatment group did during their lab time. Thecontrol group’s actual time on task per week is therefore very similar tothe treatment group’s averages.

As the semester progressed, students were able to do increasingly com-plex tasks in approximately the same amount of time, that is, 28-37 min-utes (see Table 5).

Table 5Lab Assignment: Average Time on Task and Ranking

Average time on task Average ranking of taskin minutes (range) on 1-5 scale (range)

By taskReader 43 (97) 3.5 (3.2)Dasher 36 (55) 3.4 (3.4)Video 24 (41) 4.2 (2.0)

By chapterChapter 6 37 (62) 3.8 (3.3)Chapter 7 44 (123) 3.6 (3.0)Chapter 8 34 (63) 3.5 (3.0)Chapter 9 27 (35) 3.8 (2.3)Chapter 10 30 (38) 3.9 (2.7)

Overall 34 (64) 3.7 (2.9)

Note that the range of time spent decreased from over an hour in thefirst three chapters to a little over a half hour in the final two chapters.The reason for this trend is unclear; students possibly learned to use thetechnology more efficiently or simply felt they had less time to spend onthe labs as the semester progressed. In addition, the dramatic decrease intime spent doing Dasher exercises between lab assignments 7 and 9 nodoubt resulted from our decision to turn off Dasher’s triple repetition ofitems answered wrong before giving students the correct answer. This rep-etition had proved frustrating to students and thus seemed to hamperrather than enhance learning.



STUDENTS’ EVALUATIONS OF THE TELL COMPONENTS

Over the semester, the average student ranking of the lab assignmentsby chapter was fairly constant, from 3.5 to 3.9 on a Likert scale of 1“poor” to 5 “great” (see Table 5). The range in students’ assessments oflab assignments, however, decreased slightly. The waning novelty of theTELL components as well as the general semester-end anxiety college stu-dents experience was perhaps counterbalanced by their familiarity withand enjoyment of the lab components.

Of the three component types, students preferred the video (4.2 rank-ing) to the Reader and Dasher (3.5 and 3.4 respectively). This preferencewas evident in the weekly lab reports, the semester-end questionnaire (seethe final questionnaire in Appendix D), as well as the five follow-up inter-views we conducted with treatment group students. In addition to spend-ing the least amount of time on the video, students reported that theyoften viewed the tapes with a classmate. The students explained how theircollaboration was beneficial, “Yes, we collaborated, because we are bothstrong in different areas. Working together increased how much we gotout of the lab assignments.” Another student stated, “With the videos, if Icouldn’t understand a part, then I was able to ask the other student if sheknew. Between the both of us, we could work it out together.” This spon-taneous pair work increased their ability to understand these videos. Stu-dents also found the videos to be the richest source of cultural informa-tion.

In spite of some students being “annoyed” by the precision required byDasher, seven students reported that they liked Dasher. On the final ques-tionnaire, they said, for example, that “Dasher gave an opportunity tofocus on the finer details of grammar,” and that “Dasher gave good prac-tice by example.” A few students critiqued Dasher’s identical treatment ofall errors (e.g., grammar, spelling, accents).

On the final questionnaire, students were most critical of the Reader.One or two students praised the readings for giving them “first-hand ex-perience” in French and for being “informative,” but others found themtedious. The most substantial critique made by a few students on the se-mester-end questionnaire and in the follow-up interviews was that thecurrent design of the Reader did not need to be electronic. It workedequally well with pencil and paper. If, however, the students chose to printout the Reader assignment, they were unable to access the glossary andthe cultural and grammatical notes intended to inform their reading andhelp them develop their reading strategies. Worthy of note is the fact thatthe Reader was the one multimedia component that had not been pilotedin second semester French, so it follows that changes in its design andimplementation would be required.8

We asked the treatment group students to indicate whether they felt the

290 CALICO Journal


multimedia activities were effective in helping them develop the four lan-guage skills. Similarly, we asked the control group to assess the Tuesdayin-class activities.9 More students in the treatment group indicated theythought they had developed their reading and writing skills than in thecontrol group (15 vs. 12, and 16 vs. 11, respectively). The treatment group’sperceived development of writing skills is corroborated by their highertest writing scores overall (see Table 3) and their improvement on home-work writing (see Table 2).

In both groups, 15 students reported that the lab or analogous in-classactivities had helped them develop their listening skills. In ranking thehelpfulness of language learning resources, the treatment students placedthe video higher than did the control group, perhaps because they wereable to listen to it as often as they liked in the lab, whereas the controlgroup viewed the tape no more than twice in class and had no control overits use.

When asked whether the multimedia activities were effective in helpingthem develop their speaking skills, 4 of the treatment group students said“yes;” 10 of the control group students said the Tuesday in-class activitieshad helped them do so. In their comments, treatment group students notedthat although the three multimedia components did not require speaking,their speaking improved through vocabulary building and listening to thevideos; “It didn’t really help my speech or accent but it did help my vo-cabulary,” “I don’t know, I think it [speaking] ties in with listening a lot,”and “Videos were valuable for listening and speaking.”

In addition, when asked whether the follow-up speaking activities pre-pared for Wednesday’s class were effective ways to synthesize and applywhat they had learned in the lab, 9 of the treatment group said “yes,” and5 “somewhat.” They commented, “Yes, they reinforced the ideas of thelab well,” and “Yes, we were ‘forced’ to use what we had just learned,making it stick.” So, whereas they were not required to speak French dur-ing the three TELL components, students recognized that the componentsdid help prepare them for the simulated authentic speaking classroomactivities.

Finally, the treatment group gave overwhelming support for the TELL3-day-per-week French course as an option. Fifteen of 17 students said itshould be offered, only two of whom suggested some modifications.

THE CHANGING ROLES OF THE TEACHER AND STUDENTS

Given the results of this study and its supporting academic researchbase, it would seem appropriate to institute some form of TELL-basedcurriculum for foreign language courses. However, implementing a TELL-based curriculum may be a challenging task for teachers, administrators,



and students. Instructors may expect that using multimedia will decreasetheir workload and that multimedia curricula are more innovative, therebyensuring the attention of students. Although the research appears promis-ing, some fallacies regarding TELL contribute to its largely uncontrolledand unguided use. Its informed implementation requires constant evalua-tion of using technology to learn foreign language.

Designing a TELL curriculum is a new task for most teachers but, liketraditional curricula, requires a sound general pedagogy. In addition, itdemands a sound instructional technology methodology. If an instructionalcontext combines a modified traditional classroom setting with “lab time,”then the instructor must not only develop materials for learning a foreignlanguage in the classroom but also implement technology to enhance thelearning already gained in the classroom. This combination is difficult tomanage and does not save the teacher any time in materials developmentand implementation. It is a new methodology that demands practice, plan-ning, and versatility.

Moreover, not meeting a class on a fourth day may appear to be a “laborsaver.” On the contrary, in addition to the work of teaching in the tradi-tional classroom setting, there is an extensive list of steps for the teacherwho uses a lab or computers. These steps include integrating technology-based materials with traditional materials, mastering the implementationof technology-based materials, learning how to use the hardware, trouble-shooting both hardware and software, orienting students to using the hard-ware and software, being available to students who have technologicalanxieties and difficulties, adapting technology materials that fail into a“spur of the moment” traditionally oriented lesson plan, and constantlyassessing student learning and the classroom/lab environments to ensurecontinued student success.

Nonetheless, some educators misuse TELL and view it as a panacea forcurrent administrative issues. For example, at a large midwestern univer-sity, there have been significantly increasing enrollments permitted in be-ginning level foreign language classes and a drastically decreased amountof contact time with the instructor (Musumeci, 1998). Research has clearlyshown that smaller class size is a critical factor in language development(Wiggins, 1997).10 Therefore, at every level, caution is required when imple-menting TELL.

Like teachers, students also make personal adjustments in a TELL-basedcurriculum. In the early stages, they learn to use new hardware and soft-ware and to reschedule their homework periods around available lab ornetwork time. Although students may be intrigued by the idea of “miss-ing” class time in favor of “lab” time, they are now responsible for work-ing outside the traditional classroom setting and for keeping pace with therequirements of their new curriculum.

292 CALICO Journal


Having adapted successfully to a TELL curriculum, however, studentsdo find interesting ways to use their time away from the classroom. In thisstudy, as noted above, students spontaneously engaged in collaborativelearning situations, especially when using the video. Although they mayhave had some difficulties adjusting to a TELL-based curriculum, our stu-dents proved that given the tools and the time needed, they will take theinitiative to become active language learners.

Program evaluation, both formative/summative and process-/product-oriented, is a vital aspect of building a TELL curriculum. The question isnot “should we use TELL?” but “how can we most effectively use it toenhance student learning?” We must weigh all aspects of this curriculartype before deciding on its implementation. Students must learn how touse it, and teachers must know their students and be aware that studentinput focuses the instructor on the best use of technology. In order toachieve a balanced TELL curriculum, teachers must assess the materialsfrom the students’ perspective and understand how such a curriculumwill influence student learning, not only cognitively but affectively.

Discussion

Let us summarize the findings by answering our research questions.

1. Will the TELL experimental group students perform as well as thecontrol group in listening, speaking, reading, writing, and culturalunderstanding?

Treatment group students performed as well as control group studentsin listening, and they outperformed them in reading and writing overallon the achievement tests (see Table 3). In writing homework, there was asignificant difference between the groups in the way they changed overtime; the control group’s homework writing scores decreased, and thetreatment group’s writing scores increased (see Table 2). More treatmentstudents than control group students indicated that they felt the TELLcomponents (or analogous in-class activities) had helped them developtheir writing skills. Fewer treatment group students perceived the labs ashaving helped them in their speaking, and they performed somewhat morepoorly than the control group on the final speaking test. There was, how-ever, no statistically significant difference between the groups when changeover time on the pre- and postspeaking test was taken into consideration.On both the pre- and postquiz of cultural knowledge, the experimentalgroup did better than the control group; they continued to score signifi-cantly higher, even though both groups improved considerably on thismeasure.



2. How does TELL affect student motivation and foreign languageanxiety?

Neither group changed significantly over the semester in their motiva-tion to learn French or in their foreign language anxiety. It is possible that15 weeks is insufficient time to effect a positive change in either of theselanguage learning factors.

3. How does TELL affect students’ perception of meeting the goalsthey set?

In the treatment group, 14 of 17, and in the control group 11 of 15students reported having clearly made progress toward the goals (82% &73%, respectively).

4. In what ways does TELL modify the roles of the teacher and stu-dents?

The model of three class periods plus a lab assignment does not de-crease the teacher’s work. Instead, the instructor’s energies are channeledin different directions such as evaluating, choosing, designing, adaptingsoftware, serving as consultant to students, assuring that the overall courselearning objectives are being met, and that the course is an integratedwhole. Students also take on a new role as they gain the freedom to workwhen and where they choose but also face the responsibility of doing con-siderably more work outside of class.

5. Is the TELL course an effective curricular design for achieving sec-ond language development?

After carefully monitoring and assessing language skills in listening,speaking, reading, writing, and cultural knowledge, we recognize that theTELL course was indeed effective in helping our students learn French.Its strength in this particular instructional context appeared to be in writ-ing, and its weakness in speaking. We are pleased that students were ableto apply skills from the three components—the Reader, Dasher, and thevideo—to the writing of paragraphs and longer discourse. A few studentsmentioned that they improved in their speaking skills thanks to increasedvocabulary and listening to the videos. We are confident that with a fewmodifications, such transfer to speaking in semi-authentic contexts can beincreased. We need, however, to help students make the connection moreclearly and to incorporate speaking opportunities directly into the multi-media components.

294 CALICO Journal


CONCLUSION

The following are the highlights of what we learned in this study:

1. The answer to our primary research question is that students in theTELL group performed as well as those in the control group on everyskill and on cultural understanding according to our statistical mea-sures. Students reported that they felt the multimedia tasks helpedthem in writing, reading, and listening, but less so in speaking.

2. It is indeed feasible and, for the majority of the TELL students, desir-able to replace one day in four with TELL. Fifteen of 17 studentsfavored offering the three class period plus TELL format as an option,and, despite tremendous semester-end pressures, students maintainedtheir interest in the TELL components—activities which they had tocomplete on their own time. In addition, the instructor and research-ers were satisfied with student performance as well as their attitudetoward the curriculum.

3. Treatment group writing scores on the five achievements tests as awhole were significantly higher than those of the control group. Theindependently scored early and late-semester writing showed that treat-ment students improved in their homework writing and were moti-vated to write well even on the final homework assignment.

4. An unexpected consequence was that the TELL components promotedpositive and spontaneously occurring collaboration among studentsoutside the classroom.

Let us conclude by noting some limitations of this study, followed byrecommendations for continuing the study of TELL.

1. We are generally satisfied with the revisions we made to the programevaluation design after carrying out the pilot. It would, of course, beadvantageous to have larger numbers of students or to collect dataover even longer periods. Indeed, the Department of Modern Lan-guages at CMU has extended the project to first and second semesterclasses.

2. The multimedia components that we used could be enhanced, a pro-cess constantly being done in the French program at CMU.a. The TELL component needs to integrate even more authentic,

contextualized discourse. For example, there is room for consid-erable improvement in the grammar/vocabulary exercises whichmight entail using software that is more flexible than Dasher. Ingeneral, interaction among students and between students andauthentic cultural documents in the target language could be in-creased.11



b. A further revision to the TELL treatment would be to improvethe Reader; some students saw no need for the Reader, as it iscurrently designed, to be electronic. The Reader could, for ex-ample, include links to web sites, more linguistic support (the useof which would be required in follow-up assignments) and theremoval of access to the glossary during the initial reading in or-der to promote sight-reading strategies.

c. As described above, the speaking skill needs to be integrated moredirectly with the TELL component.

We envisage numerous directions that this research of multimedia en-hanced instruction could take. With respect to students and learning is-sues, the following questions need to be addressed: What is the role ofstudents’ locus of control in where and when they use technology? (To beoptimal in the college setting, it seems that students need access to allTELL components in their rooms at any time of the day.) What is thevalue of collaborative work among students using TELL outside the class-room? How could TELL be adapted to enhance learning by targeting thevarious multiple intelligences? How could TELL components be individu-alized for remediation of students with particularly weak language skillsin language learning? And finally, can TELL be developed to track effec-tively the linguistic development and cultural understanding of students?Clearly, the examination of the integration of technology into second lan-guage teaching and learning is fertile ground for future research.

296 CALICO Journal


APPENDIX A

Lab Assignment Sheet

le français 102A/Youngs/Printemps 97 Exercice de labo # 1L’étudiant(e) Date:

Fill out this checklist for each assignment you do using multimedia or lan-guage software.

Task Where? How long? Assessment Scale: Check tools used1 (poor) to 5 (great)

Reader Fre Glossary (elec.)Chapt. 6 1 - 2 - 3 - 4 - 5 Paper Dictionary(server) Netscape

DasherModLang ServerLLRC computersSpellcheckerGrammar referenceAllons-y textbookVerb bookOther? (please list)

Additional information (you may be brief):

1. Please comment on why you chose the assessment (1-5) above.

2. How did you feel during this assignment? (upset, nervous, enthusias-tic, interested, bored...)

3. Les Devoirs (essayez d’utiliser le glossaire français):



a. A l’écrit: Qu’est-ce que vous faites pour vous préparer pour la rentrée?

b. A l’oral: Préparez un dialogue entre vous et le vendeur ou la vendeusechez la librairie. Achetez tout ce qu’il vous faut pour la rentrée.

APPENDIX B

Francophone Cultural Knowledge Quiz

Answer the questions in English, or give the French expression whereasked.1. Define “Francophone.”2. Name five Francophone countries (not France).3. Name five cities in France.4. Name denominations of French and Canadian currency.5. Name as many forms of public transportation in Paris as you can.6. What can you use, besides coins, to make a phone call in France, and

where would you go to get one?7. a. Name a French political figure.

b. Name a French film star.

c. Name a French scientist.

298 CALICO Journal


8. a. What do you say if someone says “merci” to you?

b. How do you say you’re feeling fine?

9. What are some things that French people typically have for breakfast?10. What is a croque-monsieur?

a. a drink b. an opera c. a sandwich d. a type of building11. Label the order in which the following items are typically served dur-

ing a French dinner:saladcheesecoffeemain dishdessert

12. What would you purchase at a “charcuterie?”13. Which best describes a traditional French university.

a. a complex of classroom buildings and sports facilitiesb. a sprawling suburban campusc. an urban campus, no green areas

14. Which field of study does “lettres” refer to?a. education b. fine arts c. humanities d. sciences

15. Where do most French university students live?a. in rented rooms in town b. in university dormsc. with their families

16. How are floors of a building numbered in France?17. What’s a “deux-pièces?”

a. a coin b. an apartment c. a theater d. a vehicle18. What does a green cross refer to?

a. a church b. a pharmacy c. a political party19. What are the following (city, region, country) and where are they lo-

cated? What do you associate with each one?What? Where? Association?

a. Guadeloupeb. Moroccoc. Louisianad. Normandye. Bordeaux

20. How would you describe French people to someone from your coun-try? What stereotypes do you have of French people? Why?



APPENDIX C

Oral Test Grading Grid

Picture description/comparisonVocabulary/content 5 4 3 2 1 0Structures 5 4 3 2 1 0Accomplishes the task 5 4 3 2 1 0 /15

Role playVocabulary/content 5 4 3 2 1 0Structures 5 4 3 2 1 0Level of formality 3 2 1 0

(polite)Accomplishes the task 5 4 3 2 1 0 /18

Telling a story in the pastVocabulary/content 5 4 3 2 1 0Structures 5 4 3 2 1 0Accomplishes the task 5 4 3 2 1 0 /15

OverallCommunication strategies 3 2 1 0

(asking for clarity,getting point across)

Pronunciation/fluency 3 2 1 0 /6(generally accurate pronunciation,not groping for words)

TOTAL /54

300 CALICO Journal


APPENDIX D

Final Questionnaire (for treatment group)

1. What did you like most about the multimedia activities (Dasher, reader,video)?

2. What did you like least about the multimedia activities (Dasher, reader,video)?

3. Were they effective in helping you develop the following skills inFrench? Yes or no? Please explain your answers.ReadingWritingListeningSpeaking

4. Were they effective in helping you to increase your knowledge offrancophone culture? Why or why not?

5. At times were you frustrated with the multimedia activities? If so,please tell which aspects frustrated you.

6. Did the benefits of the multimedia activities outweigh your frustra-tion? Please explain.

7. Did the multimedia activities help your self-confidence to speak, read,write and listen to French? Please explain.

8. Did you ever do parts of the lab in collaboration with another stu-dent? Yes or no (circle one)? If yes, do you feel that working togetherfacilitated your learning of French?

9. Were the speaking and writing follow-up activities done on/for Wednes-day effective ways to synthesize and apply what you had learned inthe lab? Please comment:Speaking activities:Written work:

10. Rank these language learning resources from 1 (most helpful) to 7(least helpful). Feel free to comment.

Allons-y textbookAllons-y videoDasher exercisesFrench glossary (electronic)paper dictionaryReader (electronic)Spellchecker



NOTES

1 Nieves (1994) did a pilot study exploring the use of a multimedia program inSpanish entitled Exito. The project included the development of a first-semesterSpanish course using multimedia, classroom-based activities based on the multi-media materials and assessment of the students’ language development. The re-search project highlighted that students could develop a beginning level of profi-ciency in Spanish by interacting with the multimedia program and by meetingwith a professor in small groups one period per week.2 We used the Dasher authoring system to create vocabulary and grammar exer-cises (Pusack & Otto, 1992).3 It is not our intent to promote any particular software, video, or textbook mate-rials. Since we were working at Carnegie Mellon University, we chose materialsalready adopted and tested by the Department of Modern Languages.4 Following Chamot, (personal communication, November 1995), we also col-lected information about students’ affect at the beginning and end of the semes-ter. It is beyond the scope of this article to report those findings.5 We would have preferred to administer externally rated pre- and posttests on allskills but this would have taken too much class time and would thus have beenunfair to the students. Therefore, we chose to use data from classroom tests.6 The writing was analyzed using an adapted form of the ESL Composition Profile(Glisan, 1981).7 Given the wide range of time students reported, we wonder if some studentsmay have mistakenly included the time they spent preparing the follow-up writingand speaking assignments in the early weeks.8 We chose to use the Reader because the courseware had been used successfullyand repeatedly in fourth semester French classes at Carnegie Mellon University.However, neither the courseware nor the readings themselves had been piloted insecond semester classes.9 Appendix D contains the final open-ended questionnaire administered to thetreatment group. Control group students filled out an adapted questionnaire.10 At Carnegie Mellon University, the Department of Modern Languages offersinstructors the option of capping classes at 18 students.11 In 1998-1999, CMU does not use the Reader in the first year French curricu-lum but instead requires students to work with documents on the web. Thesedocuments afford even greater authenticity and provide completely up-to-dateinformation (e.g., weather reports, news).

302 CALICO Journal


REFERENCES

Alderson, J. C., & Scott, M. (1992). Insiders, outsiders and participatory evalua-tion. In J. C. Alderson (Ed.), Evaluating second language education (pp.25-58). Cambridge, UK: Cambridge University Press.

Armstrong, K. M., & Yetter-Vassot, C. (1994). Transforming teaching through tech-nology. Foreign Language Annals, 27, 475-486.

Barnett, M. A. (1989). Writing as a process. French Review, 63 (1), 34-44.

Beauvois, M. H. (1992). Computer-assisted classroom instruction in the foreignlanguage classroom: Conversation in slow motion. Foreign LanguageAnnals, 25, 455-463.

Beretta, A. (1986). A case for field-experimentation in program evaluation. Lan-guage Learning, 36, 295-309.

Bernhardt, S. A., Edwards, P., & Wojahn, P. (1989). Teaching college compositionwith computers. Written Communication, 6 (1), 108-133.

Braegger, J. D., & Rice, D. B. (1996). Allons-y: Le Français par étapes (4th ed.).Boston, MA: Heinle & Heinle.

Brandl, K. (1995). Strong and weak student preferences for error feedback optionsand responses. The Modern Language Journal, 79, 194-211.

Brown, J. D. (1989). Language program evaluation: A synthesis of existing possi-bilities. In R. K. Johnson (Ed.), The second language curriculum (pp.222-241). Cambridge, UK: Cambridge University Press.

Brown, J. D. (1995). Language program evaluation: Decision, problems and solu-tion. Annual Review of Applied Linguistics, 15, 227-248.

Bump, J. (1990). Radical changes in class discussion using networked computers.Computers and the Humanities, 24, 49-65.

Chun, D. M. (1994). Using computer networking to facilitate the acquisition ofinteractive competence. System, 22, 17-31.

Cononelos, T., & Olivia, M. (1993). Using computer networks to enhance foreignlanguage/culture education. Foreign Language Annals, 26, 527-534.

Conrad, B. K. (1996). CALL: non-English L2 instruction. Annual Review of Ap-plied Linguistics, 16, 158-181.

Costanzo, W. (1994). Reading, writing and thinking in an age of electronic literacy.In C. L. Selfe & H. Susan (Eds.), Literacy and computers (pp. 54-73).New York: The Modern Language Association of America.

Cummins, J. (1991). Introductory remarks. In R. DeVillar & C. Faltis (Eds.), Com-puters and cultural diversity: Restructuring for school success (pp. vii-ix). Albany, NY: State University of New York Press.

Daiute, C. (1984). Can the computer stimulate writer’s inner dialogues? In W.Wrench (Ed.), The computer in composition instruction. Urbana, IL:NTCE.

DeVillar, R., & Faltis, C. (1991). Computers and cultural diversity: Restructuringfor school success. Albany, NY: State University of New York Press.



DiMatteo, A. (1990). Under erasure: A theory of interactive writing in real life.Computer and Composition, 7, 71-89.

Donato, R., Antonek, J., & Tucker, G. R. (1996). Monitoring and assessing a Japa-nese FLES program: Ambiance and achievement. Language Learning,46 (3), 497-528.

Garrett, N. (1991). Technology in the service of language learning: Trends andissues. The Modern Language Journal, 75 (1), 74-101.

Glisan, E. (1981). ESL Composition profile. In H. L. Jacobs, S. Zingraf, D. Wormuth,V. Hartfield, & J. Hughey (Eds.), Testing ESL composition: A practicalapproach (pp. 323-324). Rowley, MA: Newbury House.

Greenia, G. D. (1992). Computers and teaching composition in a foreign language.Foreign Language Annals, 25, 33-47.

Hass, C. (1989). How the writing medium shapes the writing process: Effects ofword processing on planning. Research in the Teaching of English, 23,181-207.

Hawisher, G. E. (1989). Research and recommendations for computers and com-position. In G. E. Hawser, & C. L. Selfe (Eds.), Critical perspectives oncomputers and composition instruction (pp. 44-69). New York: Teach-ers College Press.

Herron, C., & Moos, M. (1993). Electronics media in the foreign language andliterature classroom: A fusion between science and the humanities. For-eign Language Annals, 26 (4), 478-490.

Horwitz, E. K., & Young, D. J. (1991). Language anxiety: From theory and re-search to classroom implications. Englewood Cliffs, NJ: Prentice Hall.

Johnson, D. M. (1985). Using computers to promote the development of English asa second language: A report to the Carnegie Corporation. Tuscon: Uni-versity of Arizona. (EDRS:ED 278 211).

Johnson, D. M. (1991). Second language and content learning with computers:Research in the role of social factors. In P. Dunkel (Ed.), Computer-as-sisted language learning and testing: Research issues and practice (pp.61-83). New York, NY: Newbury House.

Johnson, D. M. (1992). Approaches in second language learning. White Plains,NY: Longman Publishing Company.

Kelm, O. R. (1992). The use of synchronous computer networks in second lan-guage instruction: A preliminary report. Foreign Language Annals, 25(5), 441—454.

Lafford, P. A., & Lafford, B. A. (1997). Learning language and culture with internettechnologies. In M. D. Bush, & R. M. Terry (Eds.), Technology-enhancedlanguage learning (pp. 215-262). Illinois: National Textbook Company.

Liu, M., & Reed, W. M. (1994). The relationship between learning strategies andlearning styles in a hypermedia environment. Computers in Human Be-havior, 10 (4), 419-434.

Long, M. H. (1984). Process and product in ESL program evaluation. TESOLQuarterly, 18, 409-425.

304 CALICO Journal


Lynch, B. (1990). A context-adaptive model of program evaluation. TESOL Quar-terly, 24, 23-42.

Meunier, L. E. (1994). Computer-assisted language instruction and cooperativelearning. Applied Language Learning, 5 (2), 31-56.

Musumeci, D. (1998). Mallard, a web-based application in use at the University ofIllinois (technology demonstration). Pittsburgh, PA: University of Pitts-burgh.

Mydlarski, D. (1987). Cooperative computer-assisted language learning: Is it livingup to its promise? Journal of Educational Techniques and Technologies,20 (1), 26-29.

Nagata, N. (1993). Intelligent computer feedback for second language instruction.The Modern Language Journal, 77, 330-339.

Nagata, N., & Swisher, V. M. (1995). A study of consciousness-raising by com-puter: The effect of metalinguistic feedback on second language learn-ing. Foreign Language Annals, 28, 337-347.

Neu, J., & Scarcella, R. (1991). Word processing in the ESL writing classroom. InP. Dunkel (Ed.), Computer-assisted language learning and testing: Re-search issues and practices (pp. 169-187) New York: Newbury House.

Nieves, K. A. (1994). The development of a technology-based class in beginninglevel Spanish: Experiences with Exito. (Doctoral dissertation, GeorgeMason University).

Phinney, M. (1988). Computers, composition and second language teaching. In M.C. Pennington (Ed.), Teaching languages with computers: The state ofthe art. La Jolla, CA: Althesan.

Phinney, M. (1991). Computer-assisted writing and writing apprehension in ESLstudents. In P. Dunkel (Ed.), Computer-assisted language learning andtesting: Research issues and practices (pp. 189-204). New York: NewburyHouse.

Piper, A. (1986). Conversation and the computer: A study of the conversationalspinoff generated among learners of English as a foreign language work-ing in groups. System, 14, 187-198.

Pitiyanuwat, S. (1986). On-the-job behavior of trainees as a function of their reac-tion to the training program and learning achievement. In A. Wangsotorn,A. Maurice, K. Prapphal, & B. Kenny (Eds.), Trends in language pro-gram evaluation (pp. 142-154). Bangkok: Chulalongkorn University.

Pusack, J. P., & Otto, S. (1997). Taking control of multimedia. In M. D. Bush & R.M. Terry (Eds.), Technology-enhanced language learning (pp.1-46).Lincolnwood, IL: National Textbook Company.

Raschio, R. (1990). The role of cognitive style in improving computer-assisted lan-guage learning. Hispania, 73 (2), 535-541.



Reid, J. M. (1987). The learning style preferences of ESL students. TESOL Quar-terly, 21, 87-111.

Robinson, G. L. (1989). The CLCCS CALL study: Methods, error feedback, atti-tudes, and achievement. In W. F. Smith (Ed.), Modern technology inforeign language education: Application and projects (pp. 119-134).Lincolnwood, IL: National Textbook Company.

Robinson, G. L. (1991). Effective feedback strategies in CALL: Learning theoryand empirical research. In P. Dunkel (Ed.), Computer-assisted languagelearning and testing: Research issues and practice (pp. 155-167). NewYork: Newbury House.

Roblyer, M., Castine, W., & King, F. (1988). Assessing the impact of computer-based instruction. New York: Haworth Press.

Rodrigues, D. (1985). Computers and basic writers. College Composition and Com-munication, 36 (3), 336-339.

Ross, S. (1992). Program-defining evaluation in a decade of eclecticism. In J. C.Alderson & A. Beretta (Eds.), Evaluation in second language education(pp. 167-195). Cambridge, UK: Cambridge University Press.

Snow, M. A., & Brinton, D. M. (1988). Content-based instruction: Investigatingthe effectiveness of the adjunct model. TESOL Quarterly, 22, 553—574.

Stenson, N., Downing, B., Smith, J., & Smith, K. (1992). The effectiveness of com-puter-assisted pronunciation training. CALICO Journal, 9 (4), 5-18.

Svenconis, D. J., & Kerst, S. (1995). Investigating the teaching of second languagevocabulary through semantic mapping in a hypertext environment.CALICO Journal, 12 (2/3), 33-57.

Tharp, R., & Gallimore, R. (1988). Rousing minds to life: Teaching, learning, andschooling in social context. Cambridge, UK: Cambridge University Press.

Thiesmeyer, J. (1989). Should we do what we can? In G. E. Hawser, & C. L. Selfe(Eds.), Critical perspectives on computers and composition instruction.New York: Teachers College Press.

Tremblay, P. F., & Gardner, R. C. (1995). Expanding the motivation construct inlanguage learning. The Modern Language Journal, 79 (6), 505-520.

Youngs, B. E., & Jones, C. M. (1996). CMU French Reader [Computer software].Pittsburgh, PA: Carnegie Mellon University.

Vygotsky, L. S. (1978). Mind in society: The development of higher psychologicalprocesses. Cambridge, MA: Harvard University Press.

Wertsch, J. (1979). From social interaction to higher psychological processes: Aclarification and application of Vygotsky’s theory. Human Development,22, 3-22.

Wiggins, G. (1997). Educative assessment: Designing assessments to inform andimprove student performance. San Francisco, CA: Jossey-Bass.

306 CALICO Journal


ACKNOWLEDGMENTS

We are grateful to G. Richard Tucker, Barbara Freed, Christopher M. Jones,Keiko Koda and Brian MacWhinney for their feedback on the pilot studyand their encouragement throughout this project. We are indebted to ElaineRubinstein in the Office of Measurement and Evaluation of Teaching atthe University of Pittsburgh for conducting the statistical analysis, toMichael West for his help in drafting the culture quiz, and to Mark Sanfordfor rating the writing samples. We also want to acknowledge the work ofour two undergraduate research assistants during the spring of 1997: EnaKhan and Emily Spencer.

AUTHORS’ ADDRESSES

Bonnie Adair-HauckUniversity of PittsburghDepartment of Instruction and Learning4M20 Forbes QuadPittsburgh, Pa 15260Phone: 724/935-8275Fax: 724/934-8832E-mail: [email protected]

Laurel Willingham-McLain, Associate DirectorCenter for Teaching Excellence312 Administration BldgDuquesne UniversityPittsburgh, PA 15282Phone: 412/396-1760Fax: 412/396-6577E-mail: [email protected]

Bonnie Earnest YoungsDepartment of Modern LanguagesCarnegie Mellon UniversityBaker Hall 160Forbes AvenuePittsburgh, PA 15213-3890Phone: 412/268-8050 (office); 412/621-2517 (home)Fax: 412/268-1328E-mail: [email protected]

Documents

Evaluating the Integration of Technology and Second Language