Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
"anslation Educ, nchnoLRes., 33, 167-178,2010
Debate Leaming Programfor Cultivating Critical Thinking Atritudes*
Saizo AoyAGI*i, Hirotake IsHii"i, Hiroshi SmMoDA'i, Yuto ITAMi'i"2, Hiroshi ToMIE'3"4,
Kinya KITAGAwpL'3"5 and Satoshi KAwaLHARA'3 '"t
Graduate Sbhool ofEnetgy Slrience, 1<yoto Uhiversi(y )bshido Honmaehi. Sady,o-ku, 1<yoto, 606-8501 k\7an
"'
Pnesen4 7laras instruments JopanLimited 6-24-i, ?Vlshishiiu'uku, Shinjuku, 7bkyo, l60-8366 Japan'
'3
Zeze High SbhooL 2-11-L Zeze, Otscr, Shiga, 520-08i5 Jt\)an '
'`Present
Shiga Pre12ictural Boant ojCEducation, 4-1-1. 1<yomachi, Otsu, Shiga, 520-8577JLu)an '
'SPresent,
Mbrtyama High School, 3-l2-34, Mbrlyama, Shiga, 524-O022 Japan
Received for publicatton, May 25, 2010
Debate learning is known as an effective method for cultivating critica] thinking learning, However, debate learning requires much time for application in an actual classroom, This report describes a critical thinking program that includes debate learning with an internet-based debate support system. The proposed program was applied during periods of integrated learhing in the
first semester of 2008 with 428 third grade students of a Japanese high school. Its capability of
cuttjvating critical thinking attitudes was evaluated, Results show that partieipantsl critical
thinking attitudes of objectivity and good faith were improved, but improvement of inquiry consciousness (inquisitiveness) was s]ight. '
Kky worzis : critical thinking attitudes, debate, information and communication teehnology,
integrated study
1. INTRODUCTION
Critical thinking ability is regarded as an
important educational goal in school education.
Critical thinking, primarily logical thinking, is used
when people listen to others' speeches, and when
reading texts or when discussing and expressing
their opinions. Nevertheless, it is not a synonym
for logical thinking but a package of skills and
attitudes related to thinking that is used forevaluation and solution of various dhi]y problems(Michita 2001a),
Critical thinking encompasses "generation
of
hypotheses, perspectives, questions, solutions and
plans" fbr problem-solving, reflective thinking fbrintentional review of ene's own inference process(Kusumi 1996, Ennis 1987), and open-minded
thinking based on appropriate criteria or grounds(Miyamoto et aL 1996). Considering these studies,
critical thinking is defined fbr this study as
"reasonable reflective thinking that is focusecl on
deciding what to believe or do" (Eniss 1987).
Glaser (1941) pointed out that critical thinking
attitudes are a necessary ability of critieal thinking
'This paper was originaLly published in ulon, f Educ,
7lechnoL, Xiol.33, No.4, pp.411-422 (2010)
in adclition to knowledge and skills of logical
pursuit and reasoning. Existing.・Feports of the
related literature describe that ・critical
thinking
includes emotional aspects, such 'as
attitudes or
dispositions, and cognitive aspects, such as skills
and knowledge (e.g, Eniss 1987). Spontaneous or
autonomous critical thinking is impossible without
emotional aspects of critical thinking even if
cognitive aspects of critical thinking have been
learned. Consequently, cultivation of both aspectsof critical thinking is important.
'Hirayama
(2004) pointed out that Japanesestudies of emotional aspects of critical thinking
have focused on critical thinking attitudes (e.g.,Hirayama and Kusumi 2004) and an orientation
toward critical thinking (e.g. Hirooka et al 2000).Fer this study, critical//thinki'ng,attitudes are
defined as a tendency' toward・learning ancl
application of cognitive aspects ot' critical thinking,
They・iriclude conscious tendeneies to use critical
thinking such as orientation toward critical
thinkiing, and unconscious teridencies such as
critical thinking habits. ,・・
ln additlon, the japanese//,government hasintroduced integrated learning" into elementary
schools, junior high schools, and high schools
since 2002, with the aim of "cultivating
a qualityand an ability to learn and think independently by
167
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
168
and for oneseiC autonomously determine and solve
problems," and "learning
how to learn or think,
cultivate creative and spontaneous attitudes
toward problem solving and other pursuits, and
enable a person to think independently of a way to
live" eapanese Ministry of Edueation 1999).Higuchi (2003) and Sano et al (1999) pointed outthat critical thinking education would contribute
to the aim of integrated learning. Turuda and
Yukura (2007a) also described that critical
thinking education in integrated learning is an
important task, Nevertheless, it is also noted that
few education aetivities exist for critical thinkingeducation ancl development and practice of
educational programs for cultivating critical
thinking attitudes are demanded.
The purposes of this study are the proposal,
application, and evaluation of a debate-learning
program fbr cultivating critical thinking attitudes.
The proposed method targets Japanese high school
students and uses a period for integrated study,
2, CRITICALTHINKINGMTITUDES AND DEBATE LEA RN ING
Cognitive psychological reports have describedthat people tend to have biased and selfish
thought, and tend to have difficulty in critical
thinking (Miyamoto et aL 1996), Nevertheless,listening to others' thoughts promotes objective
reflection of one's own thoughts, which supports
critical thinking even if individual thought is biased.Existing studies have supported that critical
thinking attitudes include aspects of listening to
others' thoughts (e.g., Hirooka et aZ 2000,Hirayarna and Kusumi 2004), An improvement of
an aspect of listening to others' thoughts of
critical thinking attitudes requires an actual
experience in discussion with many participants,Debate learning is therefore a discussion activity
that contributes to improvement of aspects of
critical thinking attitudes.
According to Matsumoto (1996), debate isdefined as
"a
communication form: in this form,speakers of two teams are aElocated to a pro side
and con side in re}ation to a theme. They discussthe theme, aiming at making a listener understand
the superiority of one's own tearn based on
objective evidence". It requires the logical
persuasion of participants. Therefbre, it is
regarded as an effeetive method for learninglogical thinking and public speaking skills and hasbeen practiced as debate learning in educational
eontexts (Iwasaki 2002, Suzuki 2006).
S. Ao\AGI et aL
Debate learning has the fbllowing characteristics.
First, it can be understood as a kind of discussion
game. This characteristic produces a natural
situation of critical thinking toward others'
thought. Second, participants can be allocated to
a team that is opposite to their originai position or
bias in relation to a theme. Debate participantsmust consider both sides of a theme and project tt J
and anticipate the opponents counterarguments,
which contributes to the objective reflection of
one's own thinking. However, it might be difficultfbr ,beginners or those who are not experts of
debate. Participation in support of the opposite
position will cuttivate the objective reflection of
one's own thinking. Moreover, discussion within a
limited time and fixed form, such as that of
argument, eounter-argument, and rebuttal, will
promote formatjon of opinions and logical thinking.
All charaeteristics described above woulcl
promote participants' critical thinking experiences
and contribute to cultivation of eritical thinking
attitucles, [n Japan, Kamata (2004) assessed the
cultivation of cognitive aspects of critical thinking
through debate. However, few studies haveassessed the emotional aspects of critical thinking
in Japan. Tsuruda and Yukura (2007b) reported
that students who participated spontaneously indebate learning had strong critical thinking
attitudes, However, it is possible that students
who originally had strong critieal thinking attitudes
participated in debate learning, and that the
learning itself is not the cause of the strong
attitudes. Cultivation of critical thinking attitudes
through debate learning should be evaluated
quantitatively for introduction into school
education in a more proactive manner.
Additienally, debate learning has presented
problems in its application in actual classroom
education up to now. All students should
participate in debate learning from the viewpoint
of equality 'of
educational occasion, Nevertheless,at a'maxiMum, only about eight participants can
join in one debate learning group. A typical
Japanese high school c}assroom has about 40students. Therefore, several debate learningimplementations must be done simultaneously or
sequentially. However, each debate learningexperience needs one teacher as a judge becauseJapanese students usually have no experience at
debate and cannot serve as a judge, However, the
number ofteachers is limited. Moreover, the time
allocated to integrated learning is Iimited: a mode
providing a shorter time of debate ]earningexperiences is clesired.
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
Debate Learning Program for Cu]tivating Critical Thinking Attitudes
Finally, debate learnjng allocates participants to
two mutually opposed sides, Often, students are
afraid that opposition will damage humanrelationships in a classroom even if the debate iscast as a discussion game, This fear inhibitseritical statements and thinking about opponents.
3, PROPOSALOFALEARNINGPROGRAM
This chapter describes a learning program fbr
cultivating Japanese high school students' criticai
thinking abilities. It includes 13 hr debate learning,
3.1. 0vei'viewoftheProgram
The proposed program consists of 13 hr forapplication during 14 weeks of a semester of'
Japanese high school ("Hour, or hr" means school
hour. One "heur"
is assumed as 50 min), Each
participant in the proposed program uses a
personal computer (PC) connected to the internet.
The number of the participants is not fixed;
however, anonymity prevails in debate learning ifmany students participate. Moreover, becausedebate learning is text-based, participants' typingabilities should not differ greatly, A social psychological study indicated that
human attitudes toward an action or an object are
coincident to actual action if cultivated through
real experience (Fazio 1987). Results of another
study also indicated that education related to
thinking needs not only one-sided instruction of
knowledge about a way of thinking but also
practice of thinking using that knowledge(Ruggiero 1988). Consequently, the experience of
critical thinking using knowledge with it isexpected to be effective fbr cultivating critical
thinking attitudes. Participants must learnknowledge about that before this mode of
eultivation of critical thinking attitudes,
Considering these facts, the proposed programcontains four learning phases presented in Fig. 1.
First, the text-reading phase providesfundamental knowledge; the critical thinking
practice phase provides an exercise of skills. Next,the research and presentation phase is conductedto exercise critical thinking, argumentation, ancl
information gathering. FinaTly, the debate-learning
phase cultivates critical thinking attitudes,
3.2. IiburProgramPhases
In (1) the text reading phase, participants learncritical thinking knowledge through reading a
textbook. "Kuritikaru
Sinkaron" (Michita et aZ
1999) is ehosen as the textbook because its
169
(1)Textreading4hr
(2)Criticalthinkingpractice1hr
(3)ResearchandPresentation4hr
(4)DebateI.earning4hr
Fig. 1. F[ow ofthe progrttm.
contents are comprehensible fbr high school
stuclents and its eontents are not too much fbr thefbur hours of this phase. Dealing with dailysituations requiring critical thinking with a
four-frame cartoon, it attracts students' interest, (2) The critical thinking practiee phase exercises
partieipants' critical thinking skills with exercise
papers. The exercise papers have target texts of
critical thinking and some guidance, Participants
think critically about the text, fbllow its guidance(e.g.
`fdo
not use ambiguous words" or "avoid
reasoning errors"), and write down their thoughts,
(3) In the research and presentation phase, they
conduct research, infbrmation gathering, and
argumentation exercises. Their meunings are
exercises of skills of critic'al thinking and
cultivating an appropriate attitude by listening to
others] thoughts. In the first 2 hr of this phase,they look for and choose a theme of critical
thinking in the internet er library. Then they wrap
up their own thoughts related to the theme. In thenext hour, they are divided into five-member
groups, Thereafter, they present and discuss theirthoughts about their themes in each group. Finally,each group chooses a member who will presenttheir thoughts to the whole classreom.
(4)During the debate-learning phase, they
conduct debate learning tbr critical thinking
practice and cultivation of appropriate attitudes,
Although the practice is expected to affect
cultivation of critical thinking skills, the main
objective of this phase is cultivation of critieal
thinking attitudes, During this phase, a debatesupport system is used as a solution to the
problem that debate learning requires many judgesto secure all students' participation. Detaiis of
the debate support system are described in 3.3.The first 2 hr of this phase are periods for
.research of given theme and preparation of
argumentation, The subsequent 2 hr are best
arranged as a continuous 2 hr period becausedebate learning usually requires more than 50 min.
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
170 S. AOYAGI et aL
3.3, DebateSupportSystem
The debate support system is an internet-basedenvironment fbr debate iearning developed byTerado et aZ (2005). Each participant of debatelearning uses one PC connected to the internet,Each belongs to a group of fbur members and
discusses a theme, Numbers of groups and
partieipants who can diseuss an issuesimultaneously are limited only by the available
equipment (e.g, number of PCs).
Figure 2 shows the flow of debate tearning withthe system. Arrows from
"lst
argument" to"counter
argument" show that "counter
arguments"
are inputted against the "lst
argument". Similarly,an arrow from the
"counter
argument" to"rebuttal"
and an ttrrovL, from "rebuttal"
to "2nd
counter argument" mean that the latterstatements are inputtedi against the formerstatement. First, participants input their own
opinion from their original position toward a giventheme in
"Inputting opinion". Second, participants
input a logical proposition with objeetive evidence
from their given posiLion toward the san]e theme
in the "lst
argument". A teacher or supervisor
must set the position allocation in advance at the
system control paneE. Next, participants criticize
or ask questions about the opponent partieipants'argumentation during "eounter
arguments" and
answer the opponent participants' counter
arguments during a "rebuttal",
Moreover,
participants can ask questions or criticize the
opponent participants' rebutta] again in "2nd
counter argument". The main discussion part of
The pros side The eons side
lstargument lstargument
Counterargument Counterargument
Rebuttal Rebuttal
2ndcounterargument2ndcounterargument
Fig. 2. Debate procedure using the educational debate system.
debate learning is from "argumentation"
to the"2nd
counter argument". Finally, purticipants
input their own opinion from their original position
toward the theme in "Inputting
opinion" forselFreflection of their own opinion change
through a debate learning experience, and input
some impressions or comments in C`Comments
ttnd
impressions". Figure 3 presents a screenshot of
the "counter
argument" in the system.
Debate tearning using the system presents some
important characteristics that are expected tocontribute to cultivation of critical thinking ability
in addition to the function that enables numerous
participants to join in a discussion. Usual debatelearning has a con team and a pro team; each team
member is responsible for one statement such as"`argumentation"
or "counter
argument". Incontrast, debate using the system conducts pluralone-to-one debates in parallel, For example, ifthe con team and the pro team each has two
participants, then con side participants counter
two individual pro side participants and participatein two one-to-one debates simultaneously. Onedebate has four statements for occasions fbr a
partieipant, as shown in Fig. 2. Therefbre, debatewith the sysLem has more statement opportunities
than a conventionally conducted debate, Moreover,a single participant must make all statements,
Theretbre, each participant is compelled to think.
Next, the statements are displayed on a PCmonitor when purticipants' input statements
counter the opponent's statements. Thischaracteristic allows participants to analyze
opponents' sLatemenLs more easily, Moreover,
participants can easity revise or elaborate their
own statement logical}y because all statements are
text-based.
Fig. 3. Screenshot of the counter argument step ofthe
debate leurning system,
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
Debate Learning Program for
This system provides anonymity. Participants,known solely by their ID numbers, do not worry
about human relations in the classroom, which
promotes active statements and critical thinking,
Debates with the system have no judge. This
chttracteristic constitutes an important merit
because it obviates the need for human resources
serving as judges, However, because the logic of
discussion is not evaluated by a judge, a pressurefactor that promotes participants' logicaldiscussion is lacking. For that reason, bad debatessuch as emotional battles of words are more likelyto occur. To avoid bad debates, the proposed
program teaches them in the text-reading phaseand the critical thinking practice phase what
logieal thinking should be and emphasizes the
necessity of logical discussion in clebaLe before thedebate-learning phase.
Moreover, the system interlace promotes logicalthinking. For example, the input forms of opinions,
arguments, and evidence are separated, which
makes participants understand that an argument
needs evidence ancl that it reqnires support and
that logical thinking based evidence will be
promoted.
3.4. fumitiarity ofa 77ieme andParticipation with
an CPposite Position
Familiarity of a given theme in debate learningand whether students participate in debate with
their original position to a given theme or with
opposite position (participation with opposite
position) are expected to affect cultivation of
critical thinking attitucles,
Higuehi (2003) pointed out that problemsincluding some elements of reality are suitable forcultivation of critical thinking ability because such
problems are easy to understand and easy to
imagine for participants. According to that report,
a familiar theme is expected to be suitable indebate learning fbr cultivating critical thinking
attitudes, IIowever, participants might have their
own position or opinion. This existing position or
opinion might disturb open-minded thinking, which
is related to criLical thinking.
Participants can participate in a debate with a
team that opposes their original opinion te a theme.
As described above, participation with an opposite
position will boost their objective reflection of their
o-n thinking (Iwasaki 2002). Nevertheless,
participation with an opposite position will eause
cognitive dissonance, which might cause difficulty
in debate and critical thinking.
Consequently, two conditions of familiarity of a
Cu]tivating Criticai 'i'hinking
Attitudes 171
given theme and participation with an opposite
position have both benefits and shortcomings indebate learning that cannot be resolved now. It is
necessary to reveal which condition is appropriatefor cuitivatioll ofcritical thinking attitudes through
application in a rea] educational context for
quantitative evaluation,
4. APPLICAI]IONOFTHEPROGRAM
The proposed program was appliecl duringperiods of integrated learning in the first semesterof 2008 with 428 third year students of a Japanesehigh school. The application flow, described in3.1., is presenLed in Fig, 1, Teachers who were incharge of each class supervised all phases of the
proposed program. The debate-learning phase was
also supervised by at least one uuthor, who
attended to answer questions about the debatesupport system,
Some environmental theTnes were used for
praetice during debate learning. Table 1 presents a
lisL of the themes discussed during debate learning,
4.1. ProgramEvaluationMizthod
Four phases of the proposed program presentedin Fig, 1 were intended to cultivate critical thinking
attitudes; their skills were also slated to becultivated. Nevertheless, because the proposed
program's originality depends on the
debate-learning phase, evaluation of cu]tivation of
critical thinking is set as the primary purpose of this
eva}uation during its application. Next, according to
discussion presented in 3.4,, the secondary purpose
is to reveal (A) the familiarity ofa given theme in
debate learning and (B) conditions under which
students participate in debates with their original
position to a given theme, as appropriate forcultivating critical thinking attitudes.
The research anci presentation phase is
expected to be effective to cultivate crltical
thinklng attitudes, as described in 3.2.Consequently, if critical thinking attitudes are
measured only betbre and after the
debate-Iearning phase and if the effeets at the
debate-learning phase and the research and
presentation phase are mixed, then it will bemisunderstood as an effect of only the
debate-learning phase or just past over.
A critical thinking attitude questionnairedeseribed below was therefbre administered by the
authors and a teacher in charge of each class in thisapplication (cl) before presentation in the researchand presentatien phase (pre-presentation), (c2) at
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
172
Table 1. Themes used during debate learning
S. AOYAGI et aL
related to objective and
'Japan must introduce a
'beverage container
deposit system.'The
Japanese government must introcluce an
envlronment tax,'Japan
must introduee daylight savings time.
'The Japanese government must prohibit
installation of automatic vending machines and
remove existing automatie vending maehines.'Use
of plastic bottles must be prohibited in Japan.'Japan
must promote introduction of cars thut
use gasoline blended with bio-ethanol,'Japan
must prohibit convenience stores tbom staying open late at night,'Colleetion and recycling of plastic bottles must be halted in Japan.
the beginning of the debate-learning phase
(pre-debate), and (c3) at the end of the
debate-learning phase (post-debate). The (c2)pre-debate questionnaire was administered
approximately one week after the (cl)pre-presentation questionnaire, and the (e3)post-debate questionnaire was administered about
two weeks after (c2) pre-debate questionnaire. Thepre-presentation questionnaire was administered
long before the presentation in some classes, butanswers of these classes are regarded equally to
those ofthe pre-presentation questionnaire.
The evaluation of this application employed the
Japanese critical orientation scale (Hirooka et at
2000), which was originated by D'Angelo (1971),translated by N{iyamoto et aX (1996), ancl analyzed
statistically by Hirooka ct nl (2000> as a critical
thinking attitudes questionnaire. Hirooka et al
(2000) identified three factors ot' critical thinking
attitudes, and the items of questionnaire
measuring one ofthese t'actors, This questionnaire
has 30 items that measure respondents' action
tendency, orientation, or critical thinking ability
by selrevaluation using a seven-point scale (from"not
agree at all" to "cornpletely
agree"). Onestudy fbund that this scale inc]udes self-evaluationof an action tendency or critical thinking ability,
and that these are inappropriate as a scale fororientation (Hirayama 2004), However, because
this study's definition of critical thinking attitudes
includes related unconscious tendencies such as
habiLs describcd in chapter 1, Hirooka's scale issuitable in this application.
The critical thinking attitudes questionnaire hasthree factors: objeetivity, good faith, and
inquisitiveness. Objectivity consists of items
causal thinking, such as "I
can eonstruct a logical discussion" or "I
giveweight to facts and evidence rather than social
obligations and human feelings when I decide".Good faith consists of items related to faithfu1attitude and respect for other opinions, such as
"I
try to understand opinions different from mine".
Inquisitiveness consists efitems related to inquiry,
persistence, and euriosity in thinking, such as "T
try several ways when a solution cannot work". Asdescribed in chapter 2, debate learning ischaracterized by logical discussion with objective
evidence. Therefore, items thaL measure objectivity
are expected to increase. Furthermore, debate
learning promotes listening to other people'sopinions. Consequently, debate learning was
expected to improve respect for other opinions:
answers ot' the items that measure good faith wereexpected to rise, However, inquisitiveness is a
factor related to creativity or orientation toward a
challenge. Improvement of the inquisitivenesscharacteristie requires experience of creative
praetice or challenge assuming that critical thinking
attitudes improves concomitant]y with criticai
thinking experience. Debate tearning necessitates
building a conclusion fbllewing a logical form basedon certain evidence, and does not need creativity
or new challenges. Therefbre, answers related to
inquisitiveness will not rise.
During the debate-learning phase, participantswere divided into groups with fbur or threemembers, Grouping procedures were the fo11owing:First, a questionnaire was conducted simultaneously
as a pre-debate questionnaire, which presents eight
themes for debate learning shown in Table 1.Respondents answer whether each theme is familiaror not, and whether they agree with each theme or
not. Next, they were divided into tbur conditions
before debate leaming: famthlar themq otiginal
post'tr'on; fbl7]thhr theore, opposite postb'an; not
fdmihhr theme arvigina7 positian; and not thmi))lar
theme opposhe postir'an. The groups fundamentallyconsist ofmembers under the same condition.
4.2. ResultsandDiscussion
The valid responses were 287 because there
were fbrms that lack responses to some items, Inaddition, answers of some groups whose eonditions
of (A) familiarity of a theme and (B) participationwith opposite posiLion toward themes are not
homogeneous because of adjustment ibr absentees.
Table 2 presents the number of valid responses of
four conditions of (A) familiarity of a theme and
(B) participation with opposite position toward
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
Debate Learning Program for
Table 2, Number of responses of four conditions of
familiarity and position toward themes
Cultivating Critical Thinking Attitudes 173
Table 3, Average and standard deviation of factor scores
of' critical thinking disposition seale
FamiliarthemeNot familiar theme
Pre- Pre-presentation clebatePost-debate
Original
posltlonOpposite
position
85
67
**
71
64Objectivity 43
Good faith 35.
Inquisitiveness 31
A.5 (8)44.7 (7,4) 46.5 (8
**
themes.
Before analysis, results of questionnaires were
translated to points ("not agree at all" is one point,"completely
agree" is seven points). This point was
originally an ordinal scale, psychological study
however sees summation of lnterval scale as an
orclinal scale (e,g, Toyoda 2002), This study
regards each item's points as existing on an ordinal
sea]e and the summation of each item as on an
interval scale, Moreover, an item of critical thinking
attitudes had a misprinting: 29 items were analyzed.
4.2.1, AnalysisofFkectorPoints
First, summations of points ofitems that measure
objectivity, good faith, and inquisitiveness were
calculated as factor peints of three factors of
critical thinking attitudes, The range of factor
points of objectivity was 11-77 points. tn adclition,that of good faith was 8-56, whereas that of
inquisitiveness was 7-49. Table 3 shows the
average and standard deviation of faetor points ofpre-presentation, pre-debate, and post-debate
questionnaires of critieal thinking attitudes.
Average points jncreased as the program
progressed.
Next, one-way analysis of variance (ANOVA)and Scheffe's pair comparison were conducted Lo
reveal cuttivation of three factors of critieal
thinking attitudes through the proposed program,The results of significance assessment are
presented in Table 3. ANOVA Tables of these
analyses are shown in Tables 4-6, One-wayANOVA results $howed that the factor points of
objectivity (p<O.Ol), good faith (p<O,Ol), and
inquisitiveness (p<O,05) significantly increasedthrough the proposed program. Results of
Scheffe's paired comparison showed that factor
points of objectivity and good faith were increased
between three pairs of questionnaire timings
(p<O.05), although that of inquisitiveness was not
significantly improved (p>O.05), except tbr (cl)pre-presentation and (c3) post-debate (p<O.05).The results are coincident with expectations that
objectivity and good faith would improve and that
inquisitiveness would not improve by debate
p m - -
3 (4.6) 36.1 (5.9) 37,3 (6 *
"(5.4) 31.4(5.4) 31.7(5
.8)
,5)
,5)
*: p<O,05,
**: p<O.Ol
t /t
Table 4. ANOVA tuble ofthe factor point ofobjectivity
Source ss df MSFpsubjectmeasurement
timeerror (AS)
46079.5 286 161,O
1278.4 2 639,2 34,8 O.OO
10520.3 572 18.4total 57878.1 860
Table 5. ANOVA table ofthe factor point of good faith
Source ss-df MSFpsubjectmeasurement
tlmeerror (AS)
17075.1 286 59.7
601,5 2 300.7 15.5 O.OO
11063.0 572 19.3
total 28739.7 860
THble 6. ANOVA table of the faetor point of inquiring mjnd
Source ss df MSFpsubjectmeasurement
timeerror (AS)
20367a7 286
60.6 2
5176.1 572
71.230.l
3.3 O.04
9.0total 25604.3 860
learning, as described in 4.1. However, the whole
program improved inquisitiveness, contrary to
expectatlons.
4.2.2. Analysisofltems
In this section, each items' answers wM beanalyzed. Figures 4-6 show numbers of respondents
who answerecl each point, 1-5, in the (el)pre-presentation, (c2) pre-debate, and (c3)post-debate of objectivity, good faith, and
inquisitiveness. Numbers of answers greater than 6
points increased and quantities of answers of lessthan 2 points decreased as the program progressed.
Next, Freedman test and Scheffe]s paircomparison of each pair of three questionnaire
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
174 S. AOYAGI et aL
timings, (cl) pre-presentation, (c2) pre-debate,
and (c3) post-debate were conducted to reveal
cultivation of three factors of criticat thinking
attitudes through the proposod program, The
results of significance assessment are presented inTable 7 (item numbers correspond to Figs. 4-6).Freedman test results of 7 of 11 iterns that
measure objectivity, 4 of 9 items Lhat measure
good faith ancl 1 of 7 items that measure
inquisitiveness were significant at the 1% or 5%significance level,
As described in 4.1., the change of critical
thinking attitudes through the researeh and
presentation phase possibly atTectcd that through
the debate-learning phase, [t is also possible that
participants' original attitude of critical thinking
affects critical thinking attitudes' change duringthe proposed program. Consequently, Spearman]srank-corre]ation coefficient between results of
(cl) pre-presentation, <c2-cl> improvement of
results of (c2) pre-debate compared to (cl)pre-presentation, and <c3-c2> improvement of
results of (c3) post-debate compared to (c2)pre-clebate of all items were calculated, Results of
(cl) pre-presentation and <c2-cl> improvement of
resu]ts of (c2) pre-debate cempared to (cl)pre-presentation of all items showed negative and
significant correlations (p<O,1), Next, <c2-cl>improvement ot' results of (c2) pre-debate
compared to (cl) pre-presentation have negative
significant correlations and <c3-c2> improvement
of results of (c3) post-debate eompared to (c2)pre-debate of all items also have significant
correlation (p<0,1). The range of correlations of
(cl) to <c2-cl> is from mO.32 to -O.57. Range ofcorrelations of <c2-cl> to <c3-c2> is from
-O,30
and -O.49,
Therefore, middle-strength negative
correlations exist, Moreover, resu]ts of (cl)pre-presentation and <c3-c2> improvement of
results of (c3) post-debate compared to (c2)pre-debate of nine items show a significant
correlation (p<O.5). The range of correlations isfr'om ntO,20 to O.02, To conc]ude, critieal thinking
attitudes thaL were originally high were not
improved through the proposed program to a great
degree, These results can be interpreted as ceiling -
effects that might arise because the questionnaireis only a seven-point scale and average answers of
some questionnaire items were originally high levelat (cl) pre-presentation questionnaire and (c2)pre-debate questionnaire.
Next, each item will be analyzed. First, item No.3, item No, 5, item No. 7, and item No. 12, whichmeasure objectivity and No, 10, No. 14, and No. 23,
Itemnu[nber ti[ning
*1 *2
O% 50% 100%
ll mfi2 -3 -4 emS pm6 ee7
*1]itemnumberscorrespondtetable7.
*2:(cl)pre-presentation,(c2)premdebateand
(c3)post-dcbate
Fig. 4. Summary ofanswered points of'items
whieh measures objectivity,
which measure good laiLh, were not signMcantiy
improved. Because abilities related to objectivity
and good faith are used in debate Iearning, these
results are not consistent with expectations. In
particular, it is interesting that one similar item ot'a pttir rose significantty and another did not, such
as item No, 2 and item No. 6 and item No. 14 anditem No. 26. To conclude, these were atTected byceiling effects.
Table 8 shows quuntities of significanLly
improved items whose mode value of answered
point ot' critical thinking attitudes questionnaire
was less than 4 and items whose mode value was 5and greater than 5 in (cl) pre-presentation. In
total, 12 items that were not significantly
improved and mode values of8 items among them
are greater than 5 or equal to 5. Therefbre,
original]y high items could not be improved, whichcan be a reason why some items' results of
Freedman tests were not significant.
NeverLheless, even if' the ceiling eff'ects are
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
eonsidered, theavoid biased cietermination.
- was not
improved significantly are inexp]icable, Theresu]t is difficult to in'terpret because item No.16,
"I
try to understand an opinion differentfrom my epinion." and item Ne. 26, which are
simi]ar te item No. 23, were signlficantly
improVed. However, this is presumed to be truebecause debate learning requires a kind of
biased detefiRination to address the opponent's
logic aggressiveiy to win in debate as a game,Perhaps for this reason, it was net improved
significantly. In fact, debate is not a game of
talking down to the opponent participants' logicbut a game of persuading a judge. Therefore,biased determination cannot bring v{ctory. This
promotion ef biased tieterminatien suggests a
disadvantage of debate learning with the debatesuppert system, whiclt is characterized byhaving no judge. Moreover, the results of palr comparison ineach pair of three questionnaire timings, (cl)pre-presentation, (c2) pre-debate, and (c3)post-debate show that some items whi ¢ hmeasure objectivity significantly improved only
between (cl) pre--presentation and (e2)pre-debate; other item$ signifigftntiy improvedonly between <c2) pre-debate and (c3>post-ciebate. The items which improvedsignificantly only between (cl) pre-presentationand (c2) pre-debate are item Nos. 15, 18, and
19, They attach importance to evidence and
grounds. The' items which significantly improved
only between (c2) pre-debate and (c3)post-debate ineluete ltems related te levei
iudgment such as item No. 2, those abeut
logical diseussiofi such as item No, 17, and
those about evidence and grounds such as itemNo. 6. 'l'herefore, the research and presentation
phase were unable to cultivate attitudes related
to level judgment or logical discussion. Thecombination of the research and presentation
phase and the debate-learning phase were
finallv able to cultivate these attitudes. ' Next, item No, 8 and item No. 25significantly improved; they gontributed the
significant improvement of inquisitiveness' lactor
points. In partlcular, item No. 25 refers to
grounds: improvement reflected use of grounds inthe debate-learning phase and the research and
presentation phase. However, item No. 8 isrelated to problem-solving, and the themes of
debate learning refor to some so]utions of social
problems. This item improved, presumab}y because
Debate Learning Program fot' Cultivating Critieal Thinking
Itemreason why item No. 23
-"I .
number timing
Att'it.udes' 175
O% SO% 100% -1 re 2 -3 -4 me 5 im' 6 me 7 *1:itemnurnberseorresFx)ndtotable7.
*Z:(cl)prc-presentation,(c2)pre-debateand
(c3)postLdebate
Fig. 5. Suinmary ofnnswered points of items
-,hieh measures good faith.
ILemnumber timing
*1 *2
O% 50% 100% -1 ma2 "3 -4 -5 fi;・6 va7
*1: item numhers corresponcl to tahle 7, *2: (cl)pre-presentation, (c2)pre-debatc and
<c3>post-debate
Fig, 6. Summary of answered points of' itefns wltich mcasuFesinquisitiveness.
of deep reflection about the themes presented indebate learning. {tem No, 28 and item No, 29 were
not improved signifieantly, although mode values of
answers of these it,ems were fewer than 4 points,Consequently, low improvement of more than half of
items that measure inquisitiveness reflected the
program]s charactem'stics,
The proposed progra.rr] slgnificant}y, improved
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
176 S, AOyAGl et al.
Table 7, Significunce assessment of pair comparison ofeach pair of
three qvestiunnaire timings of a critical thinking aLtitude questionnaire
T,:s= Puirsoftimings(4)gg7P
TextefitemsResults(3)(cl)and(c2)
(cl)and(c3)<c2)and(c3)
2Idonotdecideanythinginexeitedconclition,andreasonwithanimpassiveattitude.
** cl<<c3e2<<c3
3Ieonsidergoodandbadaspectsofproblems,
5IdonotbelieveanythingwithoutaLleastsemesuspicion.
6Itrytogatherallevidencethatisbothfavorableandcontrarytomyposition.
** c]<<e3c2<<c3
7Icandistinguishthingsthatarerelatedtoaproblemfromunre]atedthings. cl<c3oL"'.s=・s・.a.12SgivewejghttofautsandevidenceratherthansocialobLigationsand
humttnfeelingswhenIdecide,c2<c3
15Ipersistentlyfixonwhetherclearevidenceexistsernot. **cl<<c2cl<<c3
17Icanconstructalogicaldiscussion. * cl<<c3c2<c3
18Icheekei,eryt'uctandpieeeofevidencethat1canthinkof, **cl<<c2cl<<c3
19Ibehttvebasedongrounds. **cl<<c2cl<<e3
21Ireudbooksandhaveadetailedknowledgeofseveralfields. ** cl<<c3c2<<c3
4Idonotholdadogmaticandpersistentattitude. ** cl<<c3c2<c3
10Icanrespcctothers'opinions.
14[cuncomprorniseit'needed.
16ILrytounderstandanopiniondiffercntfrornmyopinion. ** cl<<c3e2<<c3oooagl"tr20Iacceptgoodassertion.gorsolutionsofothers.
cl<<c3
221avoidconformingtomyowntastewhen1decide. **cl<c2cl<<c3c2<<c3
23Iavoidbiaseddetermination.
26IsupportacorrectpositionevenifitisopposiLemyposition. **cl<<e2el<<c3c2<c3
30Iamawarethatmythoughtisanythingmorethanaposition.
1IquestionsomethingthatnoonecareSabout,
8ILrytosoLveprobLemsashardasIcan, *cl<<c3
11Iliketotryanythingnew,
fnE.E・・Z's:m
24Itryseveralwayswhenag.olutioncannotwork.
25Ipursueotherpossibilitiesifanassertionhasonlyweakevidence. cl<c2cl<c3
28Icontinuetopursueananswerevenifotherpeeplegiveup.
29IfollowthroughwithsomethingthntIdecide.
91donothesitatetodeeidewhenneeded. **cl<<e3c2<<c3zgo131trytoconsidernoton]yoneortwopositions,buteverypesition.
*cl<<c3c2<c3
27Jdonotmakeaconelusionderiveddirectlyfromevideneeandavoidleapsoflogic.
** cl<c3c2<<c3
(1) x<y: yis signit'icantly higher than x (p<O.05), x<<y: vis significantly higher than x(p<O,O1),
(2) Item No, 20 hus an error in its text ("assertion" was replaced by "subjectlve")
and was removed from analyses,
(3) The Freedman test results. <4) Result of Scheffe's pair comparisnn ( (cl) pre-presentaLion, (c2) pre"debate, and (c3) post"debate).
the factor points of items reflecting objectivity expectatjon,
and good faith. Although some items did not fbllowthe expectation described in 4.1., a]most a]] were 4.2.3. Analysis of7Vietnefumiliarity andCipposite
explained after ceiling effects were considered. Positions
Only one item measuring inquisitiveness was Three-way ANOVA of three factor points of
improved significantly, which supported the critical thinking attitudes questionnaire was
NII-Electronic Mbrary
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
Debate Learning Program for Cultivating Critical Thin
calculated to examine the effect on cultivation of
critical thinking attitudes of (A) famMarity of a
theme in debate learning and (B) participation withan opposite position. The considered factors were
(A) familiarity of a theme, (B) participation with
opposite position, and (C) questionnaire timings
((c2) pre-debate and (c3) post-debate). TheANOVA tables are presented as Tables 9-11. Nofactor showed a significant interaction.
Next, the answered point of each item was
analyzed using three-way ANOVA with (A)familiarlty of a theme, (B) participation with the
opposite position, and (C) questionnaire timings.No nonparametrie method exists that
corresponded to three-way ANOVA. Therefore,three-way ANOVA was conducted after the
answered point of each item was translated to a
normal score (Nakamae 2000), Results show that
item No, 5, which measures objectivity, only has asignificant interaction between (C) questionnairetimings, in particular (c2) pre-debate between (c3)post-debate, and (A) familiarity of a theme
(p<O.05). In additien, sirnple main effect analysis
of these two lactors -(A) and (C)- showed that
effects of (A) familiarity of a theme on (c2)
pre-debate and (C) questionnaire timings on (bl)familiar theme were significant (p<O.05). Table 12
portrays an ANOVA table of this analysis,
Nevertheless, the answered point of item No. 5 of
(b2) not familiar theme was higher than (bl)familiar therne at (c2) pre-debate. Furthermore,the (bl) familiar theme showed great improvementthrough debate learning; a difference of answered
point between (bl) familiar theme and (b2) not
familiar theme was decreasecl.
Consequently, no significant difference was
found of cuEttvatien ef crittcal thinking attitudes
between two conditions of (A) familiarity ofthemesin clebate learning. The increase of item No. 5 ot'
(bl) familiar theme was greater than that of (b2)the not familiar theme, However, differences were
found between (bl) and (b2) at (c2) pre-debate.Consequently, it cannot be said that (A) familiarityofa theme affected the increase of the answered
point of item No. 5.
The effect of (B) -whether given position to a
theme is opposite to one's original position- was
Table S. Summary ofmode value of questionnaire
answer at (cl) pre-presentation and results of Freedman test
king Attitudes 177
also not significant, Based on results ofthis study,
(A) familiarity ofa theme ancl (B) participation withopposite position are presumed to affect few
participants or exerted only a slight effect.
5. CONCLUSION
In this study, a program fbr cultivating critical
thinking attitudes with debate learning was
proposed and applied for high school education.
Results confirmed that the cultivation of critical
thinking attitudes through the proposed program,
except for inquisitiveness. In fact, inquisitiveness
Table 9, ANOVA table of the factor point of objectivity
with three factors
(familiarity, opposite position, and questionnaire timings)
Source ss dfMSFp
(A) familiarity (B) position (A) (B> interactionresidual (A) (B) (C) timings (A) (C) interaction CB) (C) interaction (A) (B) (C) interactionresidual (A) (B) (C>
837.9
263.7
O.131521.6
431.5
22.4
28.9
3,O
5372.0
1
1
1283
1
1
1
1283
837.9263,7
O,1111.4・131.5
22.4
28.9
3.0
19,O
82o
2312o
O.O07
O.13
O.98
o.oo0.28O,22O.69
Tuble 10, ANOVA table ofthe faetor point ofgood faith wiLh three factors
Cfamiliarity, opposite pesition, and questionnaire timings)
Source ss dfMSFp
(A) lamiliarity
(B) position (A) (B) interactionresidual (A) (B) (C)timings <A) (C) interaction (B) (C) interaction (A) (B) (C) interactionresidual (A) (B) (C)
253.8
172.2
o.o18114.7
208,1
8.5
4.3
O.3
3593.1
1
1
1283
1
1
]
1283
253,8172.2
o.o 64.0208.1
8.5
a.3
O.3
12,7
4.02.7o.o
16.4
O.7
O.3
o.o
O.047
O.10
O.99
o.ooO.41O.56O.88
Tzble 11. ANOVA tabte ofthe factor point ofinquiring
mind with three factors
(familiarity, opposite position, and questionnaire timings)
Source
n.s.
ss dfMSFp
significant
below 4above/equal to 57
items8 items
11 items 4 items
(A) familiarity (B) position (A) (B) interactionresidual (A) (B) (C) timings (A) (C) interuction (B) (C) interaetion (A) (B) (C) interactionresiduaL <A) <B) (C>
78.4
22,9
289,414207,7
10.6
2,O
O,1
15.1
2368.5
1
1
]283
1
1
1
1283
78,4
22.9289.4
50.2
10.6
2.0
O.1
15,1
8,4
1,6O.55.8
1.3O.2o.o1,8
O.21O.50O.02
O.26O.62O.91O.18
Japan Society for Educational Technology
NII-Electronic Library Service
JapanSocietyfor Educational Technology
178
Tab]e 12,(familiarity
S. AoyAGI et aL
ANOVA table of item No. 5 with three t'actors IWASAKI, M.
, opposite position, and quesLionnaire timings) Mezasumono.
Sourco ss df MSFp
<A) fami]iarity <B) position (A) (B) interactionrosidua[ (A) <B) (C) timings (A) (C) interaetion (B) (C) interaction (A) (B) (C) interactionresiduaKA) (B) (C)
1.4
o.o
o,o173.9
O.2 O.6 1,3
O.1
83,6
1
1
1283
2 2 2
2566
1.4o.oo.oO.6O.1O,3O,6o,oO.l2.3 O.13
O.O O,84
O,O O.98
O,6 O.52
2.1 O.12
4.3 O.Ol
O.3 O.72
is expected to be cultivttted by programs of
another type that require creativity of participants,
For that goal, a new program should be developed
in the future,
Familiarity of a theme in debate learning and
whether a given positjon to a theme is opposite
one's own original position have no effect,
statistically speaking, on cultivaLion of critical
thinking attitudes. Consequently, consideration of
these conditions in the proposed program isunnecessary,
REFERENCES
D'ANGELO, E, (1971) 7ho 7bachi)]g of'Critical 7)hinking. B,R. Gruner, AmsterdamENIss, R.II, (1987) A 7hxononly of' aiticel 71bth?kihg Piispositibns and Abiibke'es. Teaehing thinking skills:
Theory and practice, W,H. Freeman and Co., N.Y.,
pp.9-26I'Azlo, R,ll. and ZANpt'A, M.P, (1978) On the Predictive
Validity of Attitudes: The Role of Direct Experience and Cunfidence. fourveal offlorsonahtr, 46(2) : 228- 243GT.,xsER,
E,M. (1941) .4n IZ\petiment i)i 7]be Pei.elQpment of' dir'tical 7)5ihimg. CoLumbia University Prcss, N.Y.HIGucHI, N. (2003) Sougougakushuu ni Okeru
Hihnnnteki Shikou Ginou -Tanngenn
Kaihatsu no
Zenntei to site-, ilYssho DEigeku .gihntrkakubu Kbnnkyuu twou, 1 : 59-71 (En Japanese)HIRAYAM.A, R. (2004) A Review of the Measurement of
CriLical Thinking Disposition and ,ALbilities. Kvoto [,'hiveisity Rbseat'ch studi'es i?) Etrucntion, 50 :
290-302 (in Japanese)HIMYAMA, R. and KusuMt,
'I'. (2004) EtTect of Critieal
Thinking Disposition om lnterpretation ol'
ControveFsia] Issues: Evaluating Evidences und
Drawing Conclusions. Jopfinese ,foumitl ofEUucfftibnal
Rsychology, 52 (2) : 186-198 (in Japanese)H[RooKix, S., Oc;AwA, K. and MoToyosHI, T. (2000) A Exploratory Study ofMeasurement of
"the Orientation
toward Critical Thinking"'. iW7b Pnilrxiku Kyountu Ghkubu Kennilywti IifZpvu a'CFouiku KagekLti, 51: 161-173 (in Japanese)
(2002) Kuritikaru Sinnkinngu No
Kyoto C・OJiversity 7letsugfiku
Konfilyuusitsu KZyou Rvspectus, 5 : 12-27 (in Japanese)Japanese Ministry of Education (1999) Course of Study
ofHigh Schoo], http:11-"NTw.mext.go,jpfb-menu!shuppan/sonotaf9903 Olf03122603,htm (ncccssed 2009.4,6) (in Japanese)KAMATA, H. (2004) Kyoushoku Kamoku"Shakai Kouminnka Shidouhou" ni Mirareru Dibe-to Jugyou no Eikyou
-Hihanteki Sikou Nouryoku no Kouka Sokutei
no Kokeromi-. Daigeku KFouiku, Kyuushuu Daigziku Dafpa?u A)ouiku sevita-, 1O : 41-58 <in Japanese)Kv・suMl, T. (1996) K]??oteki' SZ"tonn to hihai]nteki' suou, iNO]7nehi S)hnagaA/u 5)kou. University of Tokyo Press,
Tokyo, Chapter 2. (in Japanese)MATSuMoTo (1996) Atnma wo ff7taeru Dideeto iNYuuman.
Koudansha, Tokyo (in Japanese)MICHITA, Y. (2001a) Various Coneepts of Critical
Thinkingi What Do 1'hey '1'hink
It Is?. Budetl?i of
Cbllege oflkfucation [,hiverst'ty of the Rvukyus, 59 :
109-127 (in Japanese)MlcHTTA, Y. (2001b) Critical Thinking of University Students in Reading Non-Academic Materials: ALtiLude and iXbiLity Differences in Relatien to
Academic I.evel and Major. 7he kmpanese .ibL"r?al of'
EUucationa/1{v.chology, 49 : 41-49 (in Japanese)MIcrHITA, Y., MlyAMo'ro, H. und ,ALKITsul, R. (1999) lkZJiit?karu 5?bnkfi- 1lonfi. Kitaohji Shobo, Kyoto (in Japanese)M[YAMOTO, Il., MIC}IITA, Y., TANIGUCHI, T., KIKLCHI, S., ZECHMEISTER, E.B. and JlloNsoN, J,E, (1996) diiticaf 7]bi)ikihg Kitaohji Shebou, Kyoto (in Japanese)NAKAMAE, M. (2000) Study of the ReLiability of Visual Evuluation by Lhe Ranking Method: Analysis of
OFdina] Sca]e and PsychologicaL Sca]ing Using the
Normalized-runk Approuch. ,Mhonn i[loushasenn
Gbbtu dekkniZlasshil 56 (5) : 725-730 (in Japanese)RUGGIERo, V.R. (1988) 7baahbzg 7]binimg across the
drnv'culam, Harper & Row, N,Y.SANO, Y,, ARATA, S., SIBANo, A. and YosHIDA, K. C1999) SbLrgoutaki' na Gakushuu no Z)kann to
Abbngtmn KbiSeil Kouyou Shobou, Kyoto (in Japanese)SLZUKL, K., OOI, K. and TiXKEMAE, H. (2006) Kurnhatv 5)bkhge to Kyowhu Athonn no Kvoiuku "o Shikoutsku Slett'u. Sekai Sisou Sha, Kyoto (in Japanese)TERADo, M., SHIMoDA, H., SEK[yAN{.A, T., TOMI・1・,X, K. and YosHIK,xwix, H. (2005) DevelopmenL of Lhe debate
supporL system and inLroduction to energy and
environmental edueation, Hiimun inieLlhce Sympostim eO05: 773-778 (in Japanese)TOI'ADA, H. (2002) Kbumaku h7mnnou Monn
-.,Nlrutunonn flen. Asakura Shotenn, Tokyo (in
Japanese)TURUDA, M. and YLJKURA, M, (2007a) Critica] Thinking
Disposition and Assertiveness of High School Students in Japan. Bulletin of' the Educational Reseurch and
Deve]epment, Flaculty of Ellucntion KZtgoshi)pa (yhi'vengicp; 17 : 235-245 (in Japanese)TuRuDA, M. and YuKuRA, M. (2007b) Koukousei no Eigo
Dibe-Lo KaLsuduu ga IIihannteki Sikou Taido Oyobi Assa-shenn Sukiru ni Oyobosu Eikyou. fapanese A,s,socihtion of tJliucational RgFT]hoiagy Sbukoi ilonnbtu]nshuu, 49 : 481 (in Japanese)