Pragmatics & Language Learning Vol. 13, pp. 245–266Tim Greer, Donna Tatsuki, & Carsten Roever (Eds.), 2013Honolulu, HI: University of Hawai‘i, National Foreign Language Resource Center

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

draf

t: S

epte

mbe

r 5, 2

014

3:27

PM

Yamato, Tagashira, & Isoda

Pragmatic awareness of Japanese EFL learners

Pragmatic Awareness of Japanese EFL Learners in Relation to

Individual Differences: A Cluster Analytic Approach

Kazuhito YamatoKobe University, Japan

Kenji TagashiraHiroshima University, Japan

Takamichi Isoda Ritsumeikan University, Japan

This study investigated the relationships between pragmatic awareness and learner types drawn from profiles of learners’ motivation and proficiency, thus providing insight into the interplay of learner factors in contrast to previous studies describing the relationships between single variables. In addition to a modified replication of Bardovi-Harlig and Dörnyei’s (1998) study in the Japanese EFL context, this study incorporated measures of motivation based on Self-Determination Theory and learners’ proficiency. Cluster analysis sorted 69 Japanese EFL learners into three distinct subgroups based on their configuration of motivation and proficiency; in particular, they differed on their levels of intrinsic motivation and proficiency. Their pragmatic awareness was later compared by use of one-way ANOVA. The results illustrated that two groups with more self-determined motivation showed sharper perception of pragmatic inappropriateness than the group with lower intrinsic motivation, even though its proficiency was higher. Based on the “noticing hypothesis” (Schmidt, 1995), we propose that intrinsically motivated learners are likely to process L2 pragmatic input at the level of ‘understanding,’ whereas

246 Yamato, Tagashira, & Isoda

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

draf

t: S

epte

mbe

r 5, 2

014

3:27

PM

those with lower motivational profiles only ‘notice the form’ but do not process it at the ‘understanding’ level. Proficiency is not in itself a sufficient condition for enabling ‘understanding.’

Introduction

In the field of Interlanguage Pragmatics (ILP), the relationship between learners’ pragmatic and grammatical awareness has been an ongoing discussion since Bardovi-Harlig and Dörnyei’s (1998) seminal study, which compared learners’ pragmatic and grammatical awareness from the perspective of the learning environment (e.g., EFL or ESL) and the learners’ overall L2 proficiency. Successors to this type of research have dealt with the same variables (e.g., Niezgoda & Röver, 2001) as well as other variables such as the length of residence (LOR) in the ESL environment (Ran, 2007; Schauer, 2006; Xu, Case, & Wang, 2009). However, to date only a few researchers have taken into account learners’ individual differences. Takahashi (2001, 2005) examined how motivational factors influence one aspect of pragmatic awareness—learners’ attention in processing the target pragmalinguistic features—during instruction. Tagashira, Yamato and Isoda (2011) investigated the relationships between overall pragmatic awareness and one type of individual difference, learners’ motivation. The current study, therefore, attempts to examine further the relationship between pragmatic awareness and the interplay between two individual difference factors: the motivational profiles of Japanese EFL learners and their proficiency.

Variables influencing pragmatic awareness

It is widely acknowledged that pragmatic awareness plays an important role in developing pragmatic competence. Kasper (1996) raised three conditions for the acquisition of pragmatic knowledge: “There must be pertinent input, the input has to be noticed [emphasis added], and learners need ample opportunities to develop a high level of control” (Kasper, 1996, p. 148). In other words, to develop pragmatic competence, learners have to notice the pragmatic information in the input and understand its function in the surrounding context, i.e., they have to be pragmatically aware. Bardovi-Harlig and Dörnyei (1998) were the first to investigate pragmatic awareness by analyzing learners’ recognition of pragmatic errors and how severely they rate such errors, and the variables that play a key role in developing pragmatic awareness have become an issue since then. The major variables are: learning environment, length of residence, motivation, and proficiency.

Effect of learning environment and length of residence on pragmatic awarenessBardovi-Harlig and Dörnyei (1998) compared US-based ESL learners’

recognition and judgment of the severity of grammatical errors and pragmatic infelicities with those of high school EFL learners and teachers of English

Pragmatic awareness of Japanese EFL learners 247

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 40 50

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

in Hungary. In that study, the participants first watched a video comprising 20 scenarios, some of which contained either grammatical or pragmatic errors, and were asked via a questionnaire to evaluate the severity of the perceived linguistic/pragmatic problems for each error when the scene played a second time. As a result, Bardovi-Harlig and Dörnyei found that the ESL learners recognized a considerably higher number of pragmatic errors than grammatical ones, whereas the EFL groups were more aware of grammatical violations than of pragmatic ones. The severity ratings for the two error types also indicated a difference in the participants’ perceptions across the two learning environments: ESL learners considered the pragmatic infelicities to be more serious, whereas EFL learners perceived the grammatical errors to be more salient.

Niezgoda and Röver (2001) replicated Bardovi-Harlig and Dörnyei’s (1998) study with ESL learners in Hawai‘i and EFL learners in the Czech Republic and obtained contrasting results: the EFL learners recognized a higher number of pragmatic infelicities than the ESL learners. The EFL learners also assigned higher severity ratings to both the pragmatic and grammatical violations than did the ESL learners. Thus, Niezgoda and Röver’s data showed that the EFL learners in their study were more aware of pragmatic infelicities than the ESL learners and also perceived them to be more serious than the ESL learners did. One agreement with the original study was that ESL learners considered pragmatic errors to be more salient than grammatical violations.

Previous studies on length of residence (LOR) (Ran, 2007; Schauer, 2006; Yamanaka, 2003; Xu et al., 2009) agree that learners’ awareness of pragmatics increases as LOR in the ESL environment increases. For example, Schauer (2006) used Bardovi-Harlig and Dörnyei’s video-and-questionnaire instrument accompanied by post hoc interviews, and reported that ESL learners increased their pragmatic awareness significantly during their stay in the ESL environment. Yamanaka (2003) found a significant positive relationship between L2 learners’ interpretation of accuracy and their degree of proficiency and LOR. Xu et al., (2009) found both LOR and overall L2 proficiency influenced L2 pragmatics significantly, with overall L2 proficiency demonstrating a stronger influence in the ESL environment.

Overall, these previous findings have shown that ESL learners recognize more pragmatic errors and rate them more severely than grammatical errors (Bardovi-Harlig & Dörnyei, 1998; Niezgoda & Röver, 2001), and that this tendency becomes more obvious as the learners’ LOR in the ESL environment increases (Schauer, 2006). More complex results, however, have been obtained for EFL learners. Bardovi-Harlig and Dörnyei (1998) found that EFL learners recognize more grammatical errors and rate them more severely, which led them to point out that language environment is the most important

248 Yamato, Tagashira, & Isoda

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

draf

t: S

epte

mbe

r 5, 2

014

3:27

PM

factor accounting for pragmatic and grammatical awareness. Niezgoda and Röver (2001), on the other hand, found no significant differences between grammatical and pragmatic error severity ratings among their EFL learners and argued that the “explanation lies in an interaction between exposure to pragmatic and grammatical input and individual learner characteristics, specifically the degree to which learners attend to input” (Niezgoda & Röver, 2001, p. 77). In Niezgoda and Röver’s study, the Czech-speaking EFL learners were advanced learners enrolled in a teacher education program and were highly motivated to seek pragmatic input in their daily lives, leading the researchers to argue that this motivation toward English language learning might have influenced their sensitivity to pragmatic errors as well as grammatical errors.

Effect of motivation on pragmatic awarenessThe importance of motivation in ILP is found in one of twelve basic questions

proposed by Kasper and Schmidt (1996) as “Do motivation and attitude make a difference in level of acquisition?” (p. 162) Their answer is as follows:

[It] is possible that intrinsic motivation (enjoyment of learning for its own sake) might be more relevant for ILP than extrinsic motivation (learning motivated by external reward), but then again intrinsic motivation might not be especially relevant because it is cognitive involvement and enjoyment rather than social involvement that is highlighted by the construct. (Kasper & Schmidt, 1996, pp. 161–162; emphasis in the original)In other words, motivational factors may play a role in pragmatic development.

However, only a few previous studies have dealt with motivation and pragmatic awareness to explicitly support their proposal.

Takahashi (2000, as cited in Kasper & Rose, 2002) was the first to examine the effects of motivation on L2 pragmatics, and she followed these efforts up in a successive study (Takahashi, 2005). These studies investigated the possible constraints that individual difference variables—in particular, learners’ motivation—have on the processing of L2 pragmatic input in pragmatic instruction. The motivation measure used in Takahashi (2005) was the questionnaire adapted from Schmidt, Boraie, and Kassabgy (1996). This questionnaire contains 47 items, which are categorized into seven subscales of motivation, such as intrinsic motivation, extrinsic motivation, personal goals, anxiety, and so on. The findings showed that the learners’ awareness of the target pragmalinguistic features in the input correlated with their motivation—in particular, intrinsic motivation—but not with their proficiency. One drawback of Takahashi’s approach toward motivation (and possibly also that of Schmidt et al.), however, is that motivational factors were derived a-theoretically through an exploratory factor analysis, which was necessary to reorganize the extensive subscales of motivation set out

Pragmatic awareness of Japanese EFL learners 249

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 40 50

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

in the questionnaire (e.g., Takahashi, 2005). Takahashi (2005) obtained nine factors, which was a different configuration from that in Schmidt et al. (1996). Although Takahashi noted “a complex interplay between learners’ motivational dispositions and their attentional targets at the pragmatic level” (p. 111), she further argued that “[o]ne can assume that learners with this motivational orientation [intrinsic motivation] perceive these pragmalinguistic forms as ones that allow them to achieve their language learning goals successfully, resulting in greater attention to these features” (p. 112) and concluded that pragmatic awareness “is associated with the learners’ motivation, in particular, their intrinsic motivation” (p. 113). This could result in simple dichotomous categories of motivation (intrinsic vs. extrinsic) and suggests that motivation needs to be understood through a more systematic motivational model.

Motivation as a developmental continuum— Self-Determination TheoryIn order to overcome the problems inherent in using a motivation

construct based on factor analysis, we prefer a theory-based psychological approach so that the notion of motivation can be captured more systematically and viewed as a developmental continuum. This study draws on Self-Determination Theory (SDT) (Deci & Ryan, 1985), which we find preferable to other motivational theories such as integrative motivation (Gardner, 1985) and the L2 motivational self system (Dörnyei, 2009). One advantage is that, rather than relying on a simple intrinsic/extrinsic dichotomy, SDT provides a detailed classification of motivation which differentiates motivation by the degree to which a person controls his or her behavior. The least autonomous, or self-determined stage of motivation is amotivation: a person has little or no intention to attempt the behavior. In contrast, the most self-determined stage of motivation is intrinsic motivation: a person is performing a behavior out of interest or enjoyment and is in a state of autonomy. Falling between them are types of externally-regulated, or extrinsic, motivation. Three different categories are postulated, each involving a differing degree of self-determination.1 External regulation refers to the least self-determined form of extrinsic motivation, including the classic instance of being controlled by external sources such as rewards or threats. A partially internalized type of extrinsic motivation, introjected regulation, exists within the person but is not considered part of the integrated self (e.g., learning English in order not to feel guilty). Identified regulation, which is the most developmentally advanced form of extrinsic motivation, involves appreciation of valued outcomes of the behavior, such as learning English in order to pursue one’s hobbies (Dörnyei, 1998; Hiromori, 2004).

These five categories can also be viewed as a developmental sequence. A person goes through the three stages of externally-regulated motivation,

250 Yamato, Tagashira, & Isoda

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

draf

t: S

epte

mbe

r 5, 2

014

3:27

PM

i.e., external regulation, introjected regulation, and identified regulation, and by gradually internalizing control of behavior eventually reaches the stage of fully-autonomous, intrinsic motivation.

Although the importance of motivation in L2 learning has been frequently addressed, few studies have dealt with the relationship between motivation and L2 pragmatic awareness, and the impact of learners’ level of self-determination (i.e., autonomous self-regulation) on pragmatic awareness or pragmatic development remains unclear. Tagashira et al. (2011) is the first study to investigate the relationship between motivation and pragmatic awareness based on SDT. Through cluster analysis, the data were analyzed from the perspective of learners’ motivational profiles in order to see how the profiles affected pragmatic awareness. The results revealed that learners’ motivational profiles influenced not only their perception of pragmatic error identification, but also their severity ratings of errors, suggesting that motivation plays an important role in learners’ pragmatic development.

Effect of proficiency on pragmatic awarenessIn studies on the effects of proficiency on pragmatic awareness, it has

generally been shown that learners with high proficiency are more aware of pragmatic information than those with low proficiency. This view is supported by studies such as Bardovi-Harlig and Dörnyei (1998), Niezgoda and Röver (2001), Yamanaka (2003) and Xu et al. (2009). Bialystok (1993) suggested that this is the case because learners with high proficiency have well-developed processing systems, thus allowing them to allocate selective attention to relevant aspects of input more efficiently (more accurately and faster) than those with low proficiency who struggle with processing basic semantic/syntactic features (House, 1996).

In contrast, Takahashi (2005), who studied the effects of proficiency and motivation on pragmatic awareness, found that there was no significant correlation between proficiency and pragmatic awareness. She suggested that the effects of proficiency could be overridden by motivation. This view was supported by her finding that pragmatic awareness correlated with motivation, but not with proficiency. However, the finding was not conclusive, not least because Takahashi’s study dealt with proficiency and motivation separately through correlational analyses. Therefore, the question is still open as to what the relative effect of the two individual difference factors on pragmatic awareness might be.

Research questions of the present study

As seen above, previous studies have tackled the relationship between pragmatic awareness and individual differences, and have obtained mixed results. One of the reasons behind these conflicting findings is that those previous studies tended to take variables as discrete, rather than

Pragmatic awareness of Japanese EFL learners 251

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 40 50

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

complex or integrated, concepts. For example, Bardovi-Harlig and Dörnyei (1998) considered learning environment (ESL/EFL) as a robust factor in pragmatic awareness, and Niezgoda and Röver (2001) found that overall L2 proficiency has a stronger effect on L2 pragmatics than LOR. Takahashi (2005) compared the effects of motivation and proficiency on pragmatic awareness, showing that motivation has a greater effect than proficiency, but treated them as separate variables. These studies have not dealt with the interaction or interplay between these variables. The issue here should be framed as follows: How does the interaction of proficiency and motivation relate to pragmatic awareness? Do more intrinsically motivated learners with lower proficiency levels recognize more pragmatic errors and rate them more severely than learners with less motivation and higher proficiency, and vice versa?

The objective of this study is, therefore, to clarify whether there is any difference in the pragmatic awareness of Japanese EFL learners due to their individual differences, more specifically, motivational profiles based on SDT and their proficiency (i.e., the TOEIC® score). Specifically, the research question is:

To what extent does Japanese EFL learners’ complex of individual differences (i.e., motivation and proficiency) influence their pragmatic awareness?

Method

ParticipantsThe participants were 153 Japanese university intermediate EFL learners

who had studied English for at least six years as a compulsory subject at school in Japan. Their learning environment was an EFL setting and all participants were native speakers of Japanese. Data from 62 participants were excluded from the analysis, because they had been in some sort of ESL environment for more than one day at some point prior to the present study. Data from a further 22 participants were also removed due to incomplete or missing information. Therefore, the analysis reported hereafter is based on data from 69 participants (10 men and 59 women) who had never been in an ESL environment (i.e., LOR=0).

Table 1 presents the participants’ background information including their proficiency levels measured by a standardized test, the TOEIC® (Test of English for International Communication), and self-evaluations of their English proficiency. The TOEIC test consists of separately timed listening and reading sections of 100 questions, each in a paper-and-pencil multiple-choice format, and reports an overall score in the range of 10 to 990. Information on the learners’ self-evaluated English proficiency was also obtained through a questionnaire administered at the time of the study, using a rating scale of 1 to 10 to self-assess the four skills (1=minimal, 10=near-native).

252 Yamato, Tagashira, & Isoda

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

draf

t: S

epte

mbe

r 5, 2

014

3:27

PM

Table 1. Participants’ English language proficiency (N=69)

min. max. M SDTOEIC® score 240 760 431.67 106.20self-assessment (10-point scale)

speaking 1 8 2.94 1.43listening 1 7 3.30 1.50reading 1 7 3.77 1.42writing 1 7 4.17 1.43

Materials and procedureTwo questionnaires were used for the present study: one was for measuring

English learning motivation, developed by Hiromori (2004, 2006) for Japanese EFL learners, and the other was for pragmatic awareness, originally devised by Bardovi-Harlig and Dörnyei (1998). The TOEIC® score was used as a proficiency measure. These instruments will be discussed in more detail below.

Language learning motivation scaleThe first instrument was an English learning motivation scale, adapted

from Hiromori (2004, 2006), based on SDT. The questionnaire contained a total of 18 items which asked whether various proposed reasons applied to the participants. The participants were asked to rate their agreement on a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree) (see Appendix and Hiromori, 2006, for details). On average, it took the participants approximately 20 minutes to respond to the questionnaire. All instructions were in their L1, i.e., Japanese.

Questionnaire for pragmatic awarenessThe second instrument was a questionnaire on pragmatic awareness,

adapted from Bardovi-Harlig and Dörnyei (1998). The original survey was made up of twenty scenes, consisting of three categories: (a) eight sentences which were grammatical but pragmatically inappropriate in the final line of the dialogues (i.e., pragmatically incorrect items), (b) eight sentences which were pragmatically appropriate, but contained grammatical errors (i.e., ungrammatical items), and (c) four sentences containing both grammatically correct and pragmatically appropriate sentences (see Bardovi-Harlig & Dörnyei, 1998, for detailed items). Following Bardovi-Harlig and Dörnyei, fourteen out of the twenty items were included for the present analysis, treating the first five items on the questionnaire as a practice block and eliminating one invalid item (see Bardovi-Harlig & Dörnyei, 1998, for a detailed explanation of the item selection).

Example of a pragmatically incorrect item (5 items)The teacher asks Peter to help with the plans for the class trip.T: OK, so we’ll go by bus. Who lives near the bus station? Peter, could

you check the bus time for us on the way home tonight?

Pragmatic awareness of Japanese EFL learners 253

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 40 50

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

P: No, I can’t tonight. Sorry. (from Bardovi-Harlig & Dörnyei 1998, p. 260)

Example of an ungrammatical item (6 items)Peter and George meet before class. They want to do something before

class starts.G: Hey, we’ve got 15 minutes before the next class. What shall we do?P: Let’s to go to the snack bar. (from Bardovi-Harlig & Dörnyei 1998, p. 261)

For administrative and practical reasons, instead of the video-taped format used in the original study, the test items were administered through a written questionnaire (as also in Xu et al., 2009). In addition, we made an alteration in the answer sheet from the original in order to overcome its shortcomings for analyzing the data. Figure 1 is an example of the original answer sheet used in Bardovi-Harlig and Dörnyei (1998).

The teacher asks Peter to help with the plans for the class trip.

T: OK, so we’ll go by bus. Who lives near the bus station? Peter, could you

check the bus time for us on the way home tonight?

P: No, I can’t tonight. Sorry.

Was the last part appropriate/correct? Yes No

If there was a problem, how bad do you think it was?

Not bad at all ： ： ： ： ： Very bad

Figure 1. Representation of the original answer sheet (Bardovi-Harlig & Dörnyei, 1998, p. 260).

In the original format, it is logically possible for the participants to check ‘Yes’ for a pragmatically inappropriate scenario when in fact they think it contains a grammatical error and vice versa. As Schauer (2006) rightly pointed out, in this format, “the researchers had to assume that when the participants indicated that there was an infelicity in a scenario, they had in fact detected the one planted by the researchers rather than identifying a ‘false error’” (p. 272). Niezgoda and Röver (2001) attempted to overcome this vagueness by administering an extra questionnaire of grammatical judgment after completing the original format. Putting the original and Niezgoda and Röver (2001)’s format together, we devised a separate item for pragmatic and grammatical appropriateness, as shown in Figure 2.

254 Yamato, Tagashira, & Isoda

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

draf

t: S

epte

mbe

r 5, 2

014

3:27

PM

Running head: PRAGAMATIC AWARENESS OF JAPANESE EFL LEARNERS 10

Figure 2. An example of the original answer sheet (Bardovi-Harlig & Dörnyei, 1998, p.260). In the original format, it is logically possible for the participants to tick ‘Yes’ for a pragmatically inappropriate scenario when in fact they think it contains a grammatical error and vice versa. As Schauer (2006) rightly pointed out, in this format, ‘the researchers had to assume that when the participants indicated that there was an infelicity in a scenario, they had in fact detected the one planted by the researchers rather than identifying a “false error”’ (p.272). Niezgoda and Röver (2001) attempted to overcome this vagueness by administering an extra questionnaire of grammatical judgment after completing the original format. Putting the original and Niezgoda and Röver (2001)’s format together, we devised a separate item for pragmatic and grammatical appropriateness, as shown in Figure 3:

Figure 3. An example of the modified answer sheet based on Bardovi-Harlig & Dörnyei (1998).

The teacher asks Peter to help with the plans for the class trip.

T: OK, so we’ll go by bus. Who lives near the bus station? Peter, could you

check the bus time for us on the way home tonight?

P: No, I can’t tonight. Sorry.

Was the last part …

(a) grammatically correct? Yes No

If your answer is no, how serious do you think it was? Not bad at all ： ： ： ： ： Very bad

(b) appropriate in this situation? Yes No

If no, how serious do you think it was? Not bad at all ： ： ： ： ： Very bad

The teacher asks Peter to help with the plans for the class trip.

T: OK, so we’ll go by bus. Who lives near the bus station? Peter, could you

check the bus time for us on the way home tonight?

P: No, I can’t tonight. Sorry.

Was the last part appropriate/correct? Yes No

If there was a problem, how bad do you think it was?

Not bad at all ： ： ： ： ： Very bad

Figure 2. Representation of the modified answer sheet based on Bardovi-Harlig & Dörnyei (1998).

In the present study, therefore, the participants were (a) asked to judge whether the last sentence in each scenario was pragmatically and grammatically correct, and, if they judged the item to be ‘pragmatically inappropriate’ or ‘grammatically incorrect,’ they were (b) subsequently instructed to rate the severity of the error on a six-point scale from 1 (not bad at all) to 6 (very bad). Note here that our focus in the present study is on the pragmatic awareness measured by the learners’ response to pragmatic appropriateness items (such as item (b) in Figure 2).

Data analysis

Following Bardovi-Harlig and Dörnyei (1998) and other successive studies, correct error identifications were scored as 1, and incorrect identifications were scored as 0. For error severity ratings, learners’ judgments on the six-point scales were recorded as a value from 1 to 6, and participants who had not detected an error in a scenario were assigned 0 for the severity rating. For all statistical analysis, the alpha level was set at 0.05.

To examine the configurations of motivation toward English language learning and proficiency, a group of multivariate statistical methods for classification was used to profile the learners based on their scores from the motivational questionnaire and the TOEIC® test. In previous studies, the relationship between learners’ pragmatic awareness and their individual differences was often analyzed by correlation: the focus of analysis was on the relationship between individual variables. The current study takes a different approach to analysis, which caters for the architecture of motivation postulated

Pragmatic awareness of Japanese EFL learners 255

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 40 50

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

by SDT, and proficiency. SDT’s motivational continuum suggests that learners show different degrees of intensity on the five motivational subtypes. This theoretical underpinning makes it necessary to analyze the relationship between pragmatic awareness and patterns of motivational factors with proficiency, i.e., learner profiles, rather than the correlational relationships among singles variables, such as pragmatic awareness, individual motivational factors, and proficiency.

In view of this requirement, the current study employed cluster analysis, a technique that has rarely been used in L2 research (Yamamori, Isoda, Hiromori, & Oxford, 2003). Based on similarities/dissimilarities of data, it sorts subjects and items into subgroups that share homogeneous characteristics (for further details, see Csizér, & Dörnyei, 2005 and Dörnyei, 2007). Of the many clustering algorithms, Ward’s method was used because it is generally regarded as efficient for retrieving homogeneous subgroups (Everitt, Landau, & Leese, 2001; StatSoft, Inc., 2010). Ward’s method is an agglomerative algorithm: The analysis starts with individual subjects as distinctive clusters, and larger clusters are formed by combining clusters with the closest characteristic subject until all the subjects are combined under one large cluster. This process is represented in a tree-like diagram called a dendrogram. To classify the participants, researchers need to decide a cut-point so that subgroups are formed below the cut-point. This decision is rather exploratory: researchers need to take into consideration changes in distances (dissimilarities) between clusters, characteristics of the resulting clusters, and the theoretical significance of the characteristics. The dissimilarity measure employed in this study was squared Euclidean distance as recommended for analysis using Ward’s method (Hair & Black, 2000).

Results

Learner profiles (motivation and proficiency)The number of meaningful clusters was decided by considering large

changes in clustering distances and the characteristics of the resulting clusters. With the aid of the dendrogram obtained from the English learning motivation scale and proficiency measure, participants were categorized into three groups (see Figure 3). To confirm the validity of the grouping, separate ANOVAs were conducted and results indicated significant overall differences in the combination of the score on the English learning motivation scale and TOEIC scores between the clusters (p<.01 for all).

256 Yamato, Tagashira, & Isoda

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

1 2 3 4 5 6 7 8 9

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

draf

t: S

epte

mbe

r 5, 2

014

3:27

PM

Cluster 2 ‘low proficiency/

intrinsically motivated and extrinsically

motivated’ (n = 31)

Cluster 3

‘low proficiency and intrinsically

motivated group’ (n = 25)

Cluster 1

‘high proficiency/ lower intrinsically motivated group’

(n = 13)

Figure 3. Dendrogram showing the classification of the participants according to the English learning motivation scale and proficiency.

As Figures 3 and 4 illustrate, the groups were named after their characteristics: cluster 1 ‘high proficiency/lower intrinsically motivated group’ (n=13, TOEIC® scores, M=573.08, SD=86.95), cluster 2 ‘low proficiency/intrinsically motivated and extrinsically motivated group’ (n=31, TOEIC® scores, M=396.61, SD=86.01),