CONCEPTUAL AND LEXICAL ASPECTS OF GESTURE: EVIDENCE FROM APHASIA

CONCEPTUAL AND LEXICAL ASPECTS OF GESTURE: EVIDENCE FROM APHASIA

U. Hadar and S. Yadlin-GedassyDepartment of PsychologyTela Aviv UniversityRamat Aviv 69978

Israel

Short Title: Gesture in aphasia

-Gesture in Aphasia-

Published in the Journal of Neurolinguistics,8,57-65, 1994Abstract

The patterns of speech-related ('coverbal') gestures were investigated in two

right-handed aphasic patients, one (PA) with a primary deficit in pre-

linguistic, conceptual processing and one (AU) with a primary deficit in

lexical retrieval. For reference, one healthy subject was also studied. Body

movement during speech was monitored and analysed with an advanced,

computerised technique (CODA 3). The physical properties and timing in relation

to speech of gestures of the right arm was used to grade gestures for

specificity of ideational content. Both patients produced a relatively large

amount of gestures, with relatively many content-bearing gestures. The patient

with primarily lexical deficits (AU) produced gestures of greater semantic

specificity than the patient with primarily conceptual deficits (PA). We

conclude that the gestures produced by the patients had different processing

origins according to locus of deficits: PA produced more gestures originating

in conceptual processing and AU more gestures originating in lexical

processing. By inference, we argue, gesture reflects an effort to facilitate

processing in the impaired components.

2


Introduction

After a fairly long history of ascribing to coverbal gestures primarily

communicative functions (clarifying messages, regulating speaking turns,

communicating attitudes, etc.), recent approaches argue that most gestures

originate in processes of language and speech production (Krauss et al., 1991;

Rime & Schiaratura, 1991). Such gestures are 'speech focused' (Freedman, 1972)

in the sense of being coordinated with either speech rhythm ('beats') or speech

content ('ideational gestures'). However, within approaches that view speech

production as the origin of most gestures, opinions still vary as to the

precise nature of this origin. For example, beats, to which a motor origin was

ascribed by Hadar (1989), were seen as originating in semantic (and sometimes

syntactic) marking by McNeill (1992). Similarly, ideational gestures were seen

as originating in 'motor cognition' by Rime (1983), in conceptual processes by

McNeill (1985) and in lexical processes by Butterworth & Beattie (1978), who

noted that ideational gestures tended to occur during a hesitation pause, just

prior to a content word. It has recently been suggested that ideational

gestures do not have a single processing origin: McNeill (1992) suggested that,

since speech processing is done in parallel, virtually every aspect of

processing may be reflected in gestural behavior. Hadar & Butterworth (in

press), who hold that processing components are fairly separable, suggested

that ideational gestures may originate either early in processing, when the

message to be conveyed is prepared for linguistic formulation ('conceptual

3


gestures'), or later, when linguistic processes are fairly advanced, on

occasions in which the retrieval of lexical items momentarily fails ('lexical

gestures').

Another controversial issue among speech productive approaches to gesture

regards their function. While Dittman & Llewellyn (1969) and Dittman (1972)

suggested that gestures dissipated excess energy, Rime (1983) suggested that

gesture facilitated the cognitive processes that underlie speech by allowing

'motor cognition' to process some of the material required for message

construction. McNeill (1985) argued that gestures operated in parallel to

speech (implying no facilitative role of any kind) while Morrel Samuels &

Krauss (1992) suggested that gestures facilitated lexical retrieval by a kind

of motor/conceptual priming. Butterworth & Hadar (1989) and Hadar & Butterworth

(in press) suggested that gesture reflected the facilitation of lexical

retrieval on conceptual, semantic and phonological levels. In their view

gesture was a byproduct of cognitive facilitation taking place in way of

imagistic activation.

Evidence from aphasia may, prima facie, be pertinent to the recent processing

approaches: if gestures are linked to linguistic and speech processes then,

surely, the disturbance of these processes should have implications for gesture

production. Indeed, aphasia studies were cited by the various approaches in

support of their ideas. Cicone et al. (1979) brought evidence which suggested

4


that gestures and speech operate in parallel: the impairment of one was

associated with impairment in the other. They claimed, for example, that

Broca's aphasics, who have a reduced fluency, also showed reduced incidence of

gesture. Similarly, McNeill (1985) also noted reduced gesture rate in Broca's

aphasia and developed further the idea that gestures and speech are processed

in parallel. Feyereisen (1983), however, showed that this reduced incidence

only held when measured against time. Measured against speech rate, Broca's

aphasics actually had higher incidence than healthy controls. Feyereisen's

(1983) observation was supported by Hadar (1991b). It appeared that, viewed in

relation to speech, the data from Broca's aphasia actually suggested a

compensatory increase in gesture production. The compensatory nature of gesture

production in Broca's aphasia was also supported by Hanlon et al. (1990), where

the production of gestures specifically facilitated the retrieval of words.

Cicone et al. (1979) also suggested that impaired semantics in speech

production was associated with impaired processing of gesture semantics. They

found that while in Broca's aphasia gestures were judged to clearly show in

their form aspects of speech content, this was not the case in Wernicke's

aphasia, where semantic processing was impaired. This point was again supported

in studies reported by McNeill (1992) who argued that paralleleity of

processing did not apply only to incidence, but also to semantics: semantic

impairment in speech was associated with semantic impairment in the shaping of

gestures. Here, again, controversy developed. Impaired processing of verbal

5


semantics may, on its own, create mis-matches between speech content and

gesture form (Feyereisen & Lannoy, 1991). It is clearly possible that gesture

production processes are intact despite a lack of congruity between gesture

form and speech content (Butterworth & Hadar, 1989).

Hadar & Butterworth (in press) suggested that ideational, content bearing

gestures may originate either early in processing, during pre-linguistic

message construction (to use Garrett's (1984) terminology) or late in

processing, during lexical search. They argued that the two kinds of ideational

gestures (conceptual and lexical) could be distinguished in a number of ways.

First, by contrast to beats, which tend to be short and simple, ideational

gestures were wide and complex. Complexity was defined as the number of linear

components making up the gesture. Second, ideational gestures originating in

conceptual processes tended to occur at phrase initial positions, while

gestures originating in lexical search tended to occur immediately before the

lexical item to which they affiliate. The reason for this was that gestures

were initiated immediately following their computation. When this happened

during conceptual processing the related linguistic structure (phrase or

sentence, on most current accounts, but see Garrett, 1984) was not yet ready

for articulation and could only be articulated later in time. Gestures of

lexical origin, on the other hand, were computed immediately prior to the

production of the searched-for word and so would start at that point in time.

Thirdly, gestures of lexical origin tended to occur during a hesitation pause,

6


just prior to a content word, while gestures of conceptual origins did not. In

both cases, the argument went, gesture reflected the facilitation of

processing, but since conceptual construction was not in a one-to-one relation

with articulation (much of it was done off line), processing difficulties did

not necessarily show in a hesitation pause (rather than low rate or inaccurate

formulation). Difficulties in lexical retrieval, on the other hand, did occur

on-line and therefore showed in a hesitation pause prior to a content word.

The above characterisation of the difference between the two types of

ideational gestures may be tested by evidence from aphasia. Conceptual

deficits, we argue, should result in compensatory increases in conceptual

gestures. They should therefore associate with a relatively high incidence of

ideational gestures occurring in the middle of speech and not necessarily prior

to content words. By contrast, lexical deficits should result in compensatory

increases in lexical gesture, that is, be associated with ideational gestures

occurring in proximity to a hesitation pause, just prior to a content word.

These points were tested in the present study.

Methods

Subjects: Subjects were two right-handed aphasic patients with CVA affecting

the left middle cerebral artery, whose CT scans showed damage in temporo-

7


parietal regions. Both patients had 12 years of education. AU was a 73 year old

woman and was 17 months post CVA at the time of the experimental session. PA

was a 58 year old man, 13 months post CVA. For reference, we also present data

taken from MA, a 59 year old, right-handed, healthy woman whose speech rate and

level of gesture production was midway between AU and PA (see 'Results' below).

The aphasic patients underwent language assessment which started about 2 months

prior to the experimental session, and continued for about one month after it.

The performance of AU on language input tests showed intact input processing.

She succeeded in 30 of 30 items in a test of phonological discrimination,

requiring the determination of whether or not two spoken words were identical.

AU succeeded in 44 of 47 items in a test requiring the matching of one of four

written words to a spoken word, where all written words were phonologically

close to the target, but only one matched it. She succeeded in 40 of 40 items

in a test requiring the matching of a picture to a spoken word. In a test of

lexical semantics, requiring the identification of the odd word out of four,

three of which belonged to the same semantic category while the fourth did not,

AU succeeded in 18 of 20 items. She performed correctly in 59 of 60 items in a

test requiring the determination of whether stimuli sentences, presented in

spoken form, were acceptable ('acceptability judgement'). Sentences could be

rejected on either syntactic ('John will leaving tomorrow') or semantic ('the

shoe painted the house') grounds. On the test for the recognition of grammar

(TROG), which is a subtle sentence comprehension test (Bishop & Byng, 1984), AU

8


succeeded in 77 of 80 items.

The performance of AU on language tests involving speech production showed

selective deficits. She repeated correctly 20 of 20 spoken words, and 17 of 20

spoken non-words. In a task requiring the production of a word which rhymes

with a spoken target AU succeeded in only 19 of 30 items. On a test of

spoonerism, requiring to exchange the first phoneme in two words (eg, respond

with 'lail mist' to the target 'mail list'), AU succeeded in 6 of 20 items. In

a test of semantic word fluency, requiring the production within 90 sec of as

many words as possible in a given category, AU produced 20 animal names, 17

place names and 16 food names. In a phonological word fluency test, which is

similar to the above test, but where the task is specified by first sound, AU

produced 10 words starting with /S/, 9 starting with /M/ and 8 starting with

/K/.

On the Boston Naming Test (Borod et al, 1980), requiring the naming of pictures

of different difficulty, AU succeeded in 36 of 60 items. For the 24 items she

failed to name in her first attempt, AU was given a phonemic cue (the first

sound of the target name) and this resulted in correct naming in 13 items. In 7

of the 11 remaining items, AU produced a semantic approximation of the target

(eg, 'handle' for 'knocker', 'bolt' for 'latch', etc.), and in the remaining

4 items she made phonemic errors on 1-2 phonemes. On a picture naming test for

highly familiar words, AU succeeded in 86 of 100 items. Eight of her 14 errors

9


were phonemic, and 6 were semantic approximations. On a sentence completion

test, requiring the production of a familiar word that completed a short spoken

sentence (eg, 'Snow usually falls in the' ... 'winter'), AU succeeded in 20 of

20 items.

The above profile of linguistic performance, which may also be seen in Table 1,

where her language performance is summed up and juxtaposed to PA's language

performance, clearly suggests that the primary deficit in AU's language

processing dwells in word retrieval: she showed no language input deficit on

either phoneme, word or sentence levels, but had clear picture naming problems.

These problems, it seems, originate in phonological processing: she performed

poorly on the rhyming test, failed to produce spoonerisms, and was weaker on

the phonological than on the semantic word fluency test. She often produced

phonemic errors in naming, and always indicated that she accessed the semantics

of the target word. Her success on the sentence completion test is also

consistent with context effects which often characterise anomia of a

phonological origin (Hadar et al., 1987).

{Table 1 about here}

The performance of PA on language input tests showed selective deficits. While

he showed normal, if less than perfect performance on phoneme and word levels,

he showed clear deficits on sentence and passage levels. On the test of

10


phonological discrimination PA succeeded in 25 of 30 items; on the test of word

picture matching he succeeded in 37 of 40 items; however, on the TROG he

succeeded on only 33 of 80 items (and 12 of the 33 successes were word-level

items). On a test that reverses the TROG, when one of 3 sentences describing a

simple scene has to be matched with a picture, PA succeeded in 17 of 30 items.

On the sentence acceptability test PA succeeded in 38 of 60 items, making both

false positive and false negative errors, as well as both semantic and

syntactic errors. On a test of associations based on knowledge of the world,

where he was asked to match one of two pictures with a target picture

('Pyramids and Palm Trees'), PA succeeded in 51 of 52 items.

PA showed deficits on all output tests, though these were more marked on

sublexical than on lexical levels. On word repetition, PA succeeded in 9 of 20

items and on non-word repetition he succeeded in 6 of 20 items, his errors

being phonemic in more than 90% of cases. In reading words he succeeded on 15

of 20 items, but in reading non-words he succeeded on 3 of 20 items, again

making primarily phonemic errors. In spontaneous speech, however, PA made only

few extensive phonemic errors: in telling the Cinderella story he produced only

6 non-words as compared with a total production of 169 words. On the Boston

Naming Test, PA succeeded in 26 of 60 items, again making primarily phonemic

errors (he failed to retrieve any word in 12 of his failures, and made phonemic

errors in 22 of his failures).

11


The above profile of linguistic performance suggests that PA's deficits

originate primarily in two stages of processing: sentence level semantics and

sublexical phonology, with lexical level processing being relatively preserved.

Thus, on input tasks, PA succeeded in over 80% of test items in phonemic

discrimination and word-picture matching, but in less than 65% of items in

acceptability judgement and less than 40% in sentence comprehension.

Nevertheless, he had well preserved semantic associations (based on knowledge

of the world). While impairment was seen in all of PA's output tasks, these

were relatively low on lexical level: his naming, word reading and word

repetition were over 40% correct, while his non-word reading and repetition

were under 30% correct. Moreover, most of PA's lexical level errors were

phonemic (though quite extensive at times).

Apparatus: Subjects's coverbal movement was monitored and analysed with a

computerised system called CODA-3. The movement monitoring system located small

lightweight prismatic markers attached to the subjects by tape, computed their

3-D position by triangulation and output this data at 300 Hz. No physical

connections between the subjects and the signal processing apparatus were

required, but a clear light-path to each marker was necessary (Mitchelson,

1975).

A 640K IBM XT microcomputer (PC) with a 20 megabyte hard disk, enhanced color

graphics and math co-processor acted as CODA-3 controller as well as user

12


control interface, loading software programs from discs and in EPROM memory to

the microprocessors of CODA-3. The PC received and stored data from CODA-3, as

well as determined such parameters of the stored data as temporal and spatial

resolution, point of reference, threshold values, etc. The PC can analyze the

stored data: display graphs of coordinates against time, coordinate against

coordinate and stick diagrams of joined markers against time. It also allows

selection by movable cursor of any point on display and can print its time

relative to onset, position relative to a selected reference, acceleration and

velocity.

To be able to obtain data from both arms, subjects were viewed from behind,

with the target markers placed on each upper arm and the back of the head

(Hadar et al., 1989a). An analogue form of the output of the marker located on

the head was stored on a 4-channel instrumentation recorder (Tandberg series

100) alongside and in synchrony with speech. Digital output was stored on the

hard disk at 100 Hz in 64K files, each file representing 13 sec. in real time.

The beginning of each file was marked on the analog record.

Procedure: An experimental session was conducted, in which subjects sat on a

classroom chair with no arms support, about 4m away from CODA-3. They sat with

their back to CODA-3, facing the interviewer who sat at a distance of about 2m

from them. A microphone was placed at chest level, about 20 cm in front of the

subject's knees, and was directed toward the subject (Hadar et al., 1991a).

13


The markers were attached and the subjects were encouraged to move their arms

and head to enhance habituation. Subjects were told that the experiment was

designed to monitor their movements during an interview. They were asked to

offer generous verbal responses to the interviewer's questions. An interview

was then conducted on low emotionality issues, where subjects were asked

informative questions about their work, family, hobbies etc. Interviews lasted

20 to 30 minutes.

Data Analysis: For analysis, movement in one plane was plotted against time on

the computer's monitor. A preliminary investigation (Hadar, 1991a) showed that

movement as reflected in the X coordinate (horizontal plane) highly correlated

with that of the Y axis (vertical plane) and was usually of greater amplitude.

Therefore the X coordinate was chosen for the purpose of this research. A

vectorial component (VC) was defined as the segment of movement bounded by 2

extrema (points of change of direction, seen in the chart as a peak or a

trough). A movement was defined as a set of VCs bounded by, but not containing,

VCs of smaller velocity than 10 mm/sec. This value was found to distinguish

voluntary movement from involuntary postural sway in preliminary studies

(Hadar, 1991a).

Analyses were performed on a sample of 4.2 min, stored on 20 successive

computer files, each covering 13 sec of conversation. Files selected contained

14


periods of listening whose cumulative duration was less than 5 sec, that is,

they consisted of at least 60% speaking time. Across files, speaking time was

at least 80%. Potentially ideational movements (PIMs) were defined as right-arm

gestures having at least one VC of greater amplitude than 10 mm (and, from the

definition of 'movement', of greater velocity than 10 mm/sec). This value was

found to distinguish narrow from wide movements (Hadar, 1991a). The 4.2 min

records produced an average of 30-40 PIMs, that is, the incidence of PIM was 7-

10 per min.

The recorded PIMs were sub-divided into two groups: 1) those in which the

movement pattern of the right arm was different from that of the left arm or

the head and consisted of two or more VCs; 2) those which showed similar

patterns between the left and the right upper arms, and/or possessed only one

VC. These criteria were considered to represent the distinctness and complexity

of PIMs (respectively). In accordance with the considerations of the

Introduction (above), the distinct and complex PIMs (group 1) were thought

highly probable of having ideational content. They were therefore called

'ideational gestures' (IGs).

The speech of the subjects during the 4.2 sec sample was transcribed on paper

from tape recordings of the interview. The transcription included false starts,

truncated words, hesitation pauses and their duration, etc. As a result it was

possible to reconstruct not only the text of the subjects' talk, but also the

15


related prosodic and phrase structure (boundary, continuity, repetition and

other phrase level parameters).

The timing of IGs in relation to speech was determined and classified into one

of the following categories: a- 'lexical' were IGs which started during a pause

just prior to a content word (verb, noun, adjective, etc.); b- 'conceptual'

were IGs starting during fluent speech or during a pause adjacent to a function

word; c- 'indeterminate' were IGs which could not be classed in any of the

above (e.g., during a pause followed by a circumlocution).

Results

In the following account, the number of degrees of freedom for computing

significance values is assumed to be 1, unless otherwise stated. The first set

of research hypotheses concerned the rates of movement of patients relative to

the control subject. The motivation for choosing MA as control was that her

verbal production was midway between PA and AU and overall incidence of

movement was similar to that of PA and AU, as seen in a previous study (Hadar,

1991b). Indeed, the total verbal production over the sampled data was 522 words

in MA, 715 in PA and 401 in AU. The total number of words was greater in PA

than in both of the other subjects [X2(PA,MA)= 30.11, p<0.001; X2(PA,AU)=88.35,

p<0.001] and smaller in AU than in MA [X2(AU,MA)=15.86, p<0.005]. This is

16


clearly consistent with patients' diagnosis (in terms of locus of language

deficit). The incidence of movements, across body parts and planes, was 73.8

VCs/min in MA, 75.9 VCs/min in PA and 83.6 VCs/min in AU. Since incidence

(Hadar, 1991b) ranged from 43.4 to 107.6 VCs/min in a group of 6 healthy

controls (mean=70.6 VCs/min), our subjects were well within the normal range,

though somewhat above the mean. Specifically, the incidence of right-arm VCs in

the X axis was 71.7 VCs/min for PA, 102.1 VCs/min for AU and 76.9 VCs/min for

MA. These values were again mid-range, since the 6 healthy controls gave a

range of 41.2-114.7 VCs/min in the X plane of the right arm. Here again, the

present subjects showed somewhat higher incidence than the mean of the controls

(71.2 VCs/min). At the above rates, the total right-arm production over the

sampled data ('total number of VCs' in Table 2) showed greater values for AU

than for the other two subjects, who were not different between them

[X2(AU,MA)=14.01, p<0.001; X2(AU,PA)=21.12, p<0.001; X2(MA,PA)=0.74, NS].

{Table 2 about here}

We now wanted to see whether the picture would change when short movements

(probably beats) were taken out, leaving only movements that were potentially

ideational (PIMs). As can be seen in Table 2, the total number of PIMs largely

reflected the total production rates, with AU showing higher incidence than

both of the other subjects, who were not different between them

[X2(AU,MA)=5.88, p<0.025; X2(AU,PA)=5.03, p<0.025; X2(MA,PA)=0.03, NS]. However,

17


the complexity of PIMs (measured by VCs per movement) was markedly higher in PA

(2.8 VCs/PIM) than in either AU (1.7 VCs/PIM) or MA (1.8 VCs/PIM). This was

reflected in the percentage of VCs-in-PIMs in total VCs, which was higher in PA

than in both of the other subjects, who were not different between them

[X2(PA,MA)=6.45, p<0.025; X2(PA,AU)=5.06, p<0.025; X2(AU,MA)=0.09, NS]. The

combined effect of highly complex PIMs in PA and high incidence PIMs in AU was

that the total production of right-arm movement ('number of VCs-in-PIMs' in

Table 2) was lower in the control subject than in both patients, who were not

significantly different between them [X2(AU,MA)=7.94, p<0.005; X2(PA, MA)=14.51,

p<0.001; X2(AU,PA)=1.02, NS].

The rate of gesture production relative to linguistic production (measured in

VCs-in-PIMs per word, Table 2) was higher in the anomic patient (AU) than in

both other subjects, who were not significantly different between them

[X2(AU,MA)=24.60, p<0.001; X2(AU,PA)=17.41, p<0.001; X2(PA,MA)=1.77, NS]. The

ratio of VCs in PIMs per minute was higher in both aphasic patients than in MA

[X2(PA,MA)=7.94, p<0.05; X2(AU,MA)=14.54, p<0.01], but not significantly

different between the patients [X2(AU,PA)=1.02].

Following the above investigation, data related to putative semantic and

pragmatic functions of gestures were analysed according to both the physical

features of movement and its timing relative to speech. This data is presented

in Table 3. AU had significantly more IGs than both the other subjects, but MA

18


had more IGs than PA [X2(AU,MA)=5.33, p<0.025; X2(AU,PA)=15.36, p<0.001;

X2(MA,PA)=2.81, NS]. This was reflected in the percentage of IGs in PIMs, which

was significantly higher in both AU and MA than in PA [X2(AU,PA)=6.90, p<0.005;

X2(MA,PA)=5.66, p<0.01] and similar between AU and MA [X2(AU,MA)=0.06, NS].

Table 3 also shows that AU had the greatest number of lexical IGs, whereas PA

had the smallest and MA an intermediate number [X2(AU,MA)=11.52, p<0.001;

X2(AU,PA)=24.03, p<0.001; X2(MA,PA)=3.77, p<0.06]. These differences become even

more marked when the proportion of lexical in total IGs is examined

[X2(AU,MA)=8.69, p<0.005; X2(AU,PA)=24.39, p<0.001; X2(MA,PA)=4.91, p<0.05].

Since the number of indeterminate IGs was very small, the picture regarding

conceptual IGs is the inverse of the above, with PA showing the highest

proportions, AU the lowest and MA intermediate proportions (Table 3).

Discussion

Given that the overall level of movement of our subjects was within the normal

range, though somewhat above average, the crucial data regarding the research

hypotheses concerned the intrinsic distribution of these movements. The picture

regarding PIMs was rather mixed: AU showed higher incidence, but this only

reflected her overall higher level of movement. PA, on the other hand, had

distinctly more complex PIMs, so that when the level of production of PIMs was

19


measured in number of VCs, both patients gave higher values than our control.

Although this pattern was sustained when VCs-in-PIMs were measured against

time, we must conclude that the data regarding incidence and proportion of PIMs

did not show a clear pattern of dissociations. The picture changed, though,

when PIMs were measured against linguistic production. Here AU showed markedly

higher proportion of PIMs per word than both of the other subjects, which

offered the first clear indication of a relationship between problems in word

retrieval and the production of PIMs. This indication was amplified and

specified as the ideational nature of the movements was focused upon.

AU had a greater number (and a higher proportion relative to PIMs) of IGs than

both of the other subjects, and MA showed greater values than PA. This suggests

that difficulties in word retrieval are more likely to generate IGs than

conceptual difficulties, an issue which remained open in Hadar & Butterworth

(in press), but is in line with the position held by Butterworth & Beattie

(1978) and Butterworth et al. (1981). Compatible with the present hypotheses is

the relative incidence of lexical and conceptual IGs: AU produced a markedly

higher proportion of lexical IGs than PA, while the converse held for

conceptual IGs, with MA showing intermediate proportions. This offers probably

the strongest support for the thesis of differential origins for the two kinds

of ideational gestures, as well as to the thesis of processing facilitation.

We find it remarkable that MA held intermediate values along the lexical-

20


conceptual continuum. This implies, in the present line of thought, that she

produced both lexical and conceptual gestures, each facilitating the processes

in which they respectively originate. While a single control subject does not

determine a norm, MA was matched with our two patients and gives a combination

of results that supports the internal validity of her choice as control.

The mechanism through which ideational gestures facilitate processing is not,

of course, constrained by the present results. We still have the possibility of

either motor-semantic priming (Morrel Samuels & Krauss, 1992) or imagistic

activation (Hadar & Butterworth, in press) or both. The idea of motor-semantic

priming has the advantage of building upon a mechanism which is well researched

in the study of lexical processing, namely, semantic priming (e.g., Forester,

1990). This idea is supported by findings showing that action verbs are

particularly likely to be accompanied by gestures (Krauss, 1992), though more

data are required to confirm this observation. However, there are some

difficulties with the motor-semantic priming hypothesis. Firstly, it is not

clear which kind of representation actually acts as a prime. There are two

possibilities here, one of abstract motor schemata (Schmidt, 1975) and one of

kinesthetic/proprioceptic representations. The crucial evidence for deciding

between these possibilities is whether the gesture actually has to be performed

for facilitation to occur: if it does, than the likely prime is

kinesthetic/proprioceptic, while if it does not, the likely prime is the

abstract action schema. The relevant evidence, gleaned primarily from studies

21


on the effects of immobilization on the content of speech, is still

inconclusive (Rime & Schiaratura, 1991). Secondly, it is not obvious how

motor/semantic priming could apply to lexical gestures, which are probably

triggered after the semantic selection has been performed (Butterworth & Hadar,

1989). Indeed, Rime (1983) ascribes the activation of motor-semantic

representations a primarily conceptual, probably pre-linguistic, function.

Still, it is possible that gestures are only triggered on those cases in which

the lexical failure initiates a search for a different word. In this case,

gesture could facilitate the semantic-lexical selection of the substitute word.

The testing of this possibility could be attempted if a way could be found to

separate retrieval failures which force semantic re-selection from those that

do not. It could then be determined whether cases in which the retrieved word

was different from the original target correlate better with ideational

gestures.

Imagistic activation, as Hadar & Butterworth (in press) suggest, could act

post-semantically as well as conceptually, especially if it transpires that in

selected cases images may be associated with phonological forms. In this

approach, gesture is an artifact of imagistic activation (just as lip movement

is an artifact of reading) and therefore immobilisation should not affect

either the content or the lexical fluency of speech. Also, this mechanism

should equally implicate action verbs, shape names and concrete (object) names,

so failure in selection of any of these should have equal probability of

22


producing a gesture.

Evidence from brain damaged person could help decide between these competing

mechanisms. If facilitation involves motor-semantic priming, then disorders

affecting gestures, such as limb apraxia (Feyereisen & Lannoy, 1991), should

also affect word retrieval in spontaneous speech (by increasing the length of

hesitation pauses). However, word search in patients with selectively impaired

visual imagery (see, for example, Kosslyn, 1990) should not be affected. The

converse, of course, should hold if the mechanism producing gestures is

mediated by visual imagery. Future research along these lines may help resolve

this issue.

23


References

Bishop, D.V. & Byng, S. (1984) Assessing semantic comprehension: methodological

considerations and a new clinical test. Cognitive Neuropsychology, 1, 233-244.

Borod, J.C., Goodglass, H. & Kaplan, E. (1980) Normative data on the Boston

Diagnostic Aphasia Examination, parietal lobe battery and the Boston Naming

Test. Journal of Clinical Neuropsychology, 2, 209-215.

Butterworth, B. & Beattie, G. (1978) Gesture and silence as indicators of

planning in speech. In R. Campbell & P.T. Smith (eds.), Recent Advances in the

Psychology of Language: Formal and Experimental Approaches, pp 347-360. London:

Plenum.

Butterworth, B., Swallow, J. & Grimston, M. (1981) Gestures and lexical

processes in jargon aphasia. In J.W. Brown (ed.), Jargonaphasia, pp 113-124 .

New York: Academic Press.

Butterworth, B. & Hadar, U. (1989) Gesture, speech and computational stages.

Psychological Review, 96, 168-174.

Cicone, M., Wapner, W., Foldi, N., Zurif, E. & Gardner, H. (1979) The relation

between gesture and language in aphasic communication. Brain and Language, 8,

24


324-349.

Dittman, A.T. (1972) The body movement-speech rhythm relationship as a cue to

speech encoding. In A.W. Siegman & B. Pope (eds.), Studies in Dyadic

Communication, pp 135-152. New York: Pergamon Press.

Dittman, A.T. & Llewelyn, L.G. (1969) Body movement and speech rhythm in social

conversation. Journal of Personality and Social Psychology, 11, 98-106.

Feyereisen, P. (1983) Manual activity during speaking in aphasic subjects.

International Journal of Psychology, 18, 545-556.

Feyereisen, P. & de Lannoy, J.-D. (1991) Gestures and Speech: Psychological

Investigations. Cambridge: Cambridge University Press.

Forester, K.I. (1990) Lexical processing. In D. Osherson (ed.) Invitation to

Cognitive Science I: Language. Cambridge (MA): MIT Press, pp 95-131.

Freedman, N. (1972) The analysis of movement behavior during the clinical

interview. In A.W. Siegman & B. Pope (eds.) Studies in Dyadic Communication.

New York: Pergamon Press, pp 153-175.

Garrett, M.F. (1984) The organization of processing structure for language

25


production: Applications to aphasic speech. In D. Caplan, A.R. Lecours & A.

Smith (eds.), Biological Perspectives on Language, pp 172-193. Cambridge

(Mass): MIT Press.

Hadar, U. (1989) Two types of gesture and their role in speech production.

Journal Of Language and Social Psychology, 8, 221-228.

Hadar, U. (1991a) Body movement during speech: Period analysis of upper arm and

head movement. Human Movement Science, 10, 419-446.

Hadar, U. (1991b) Speech-related body movement in aphasia: Period analysis of

upper arm and head movement. Brain and Language, 41, 339-366.

Hadar, U. & Butterworth, B. (in press) Iconic gesture, imagery and word

retrieval in speech.

Hadar, U., Jones, C. & Mate-Kole, C. (1987) The disconnection in anomic aphasia

between semantic and phonological lexicons. Cortex, 23, 505-517.

Hanlon, R.E., Brown, J.W. & Gerstman, L.J. (1990) Enhancement of naming in

nonfluent aphasia through gesture. Brain and Language, 38, 298-314.

Kosslyn, S.M. (1990) Mental imagery. In D. Osherson, S.M. Kosslyn & J.M.

26


Hollerbach (eds.) An Invitation to Cognitive Science, Vol 2: Visual Cognition

and Action, pp 73-97. Cambridge (Mass): MIT Press.

Krauss, R.M. (1992) The integration of verbal and nonverbal communication: Some

functions of conversational hand gestures. Paper presented at the joint meeting

of the European Association for Experimental Social Psychology and the Society

for Experimental Social Psychology. Louvain, Belgium.

Krauss, R.M., Morrel-Samuels, P. & Colasante, C. (1991) Do conversational hand

gestures communicate? Journal of Personality and Social Psychology, 61,

743-754.

McNeill, D. (1985) So you think gestures are nonverbal? Psychological Review,

92, 350-371.

McNeill, D. (1992) Hand and Mind: What Gestures Reveal about Thoughts. Chicago:

University of Chicago Press.

Mitchelson, D.L. (1975) Recording of movement without photography. In D.W.

Grieve, D.I. Miller, D.L. Mitchelson, P. Paul & A.J. Smith (eds.) Techniques

for the Analysis of Human Movement. London: Lepus Books, pp 33-69.

Morrel-Samuels, P. & Krauss, R.M. (1992) Word familiarity predicts the temporal

27


asynchrony of hand gestures and speech. Journal of Experimental Psychology:

Learning Memory and Cognition, 18, 615-622

Rime, B. (1983) Nonverbal communication nonverbal behavior? Towards a

cognitive-motor theory of nonverbal behavior. In W. Doise & S. Moscovici

(eds.), Current Issues in European Social Psychology, pp 85-141. Cambridge:

Cambridge University Press.

Rime, B. & Schiaratura, L. (1991) Gesture and speech. In R.S. Feldman & B. Rime

(eds.), Fundamentals of Nonverbal Behavior. Cambridge: Cambridge University

Press, pp 239-284.

Schmidt, R.A. (1975) Schema theory of discrete motor skill learning.

Psychological Review, 82, 225-260.

28


Table 1: Summary of performance of AU and PA on those language tests which they

both performed, presented as number of correct responses/total number of test

items and, in brackets, % correct of total items.

Test AU PA

Phonological discrimination 30/30 (100%) 25/30 (83.3%)

Word-picture matching 40/40 (100%) 37/40 (92.5%)

Acceptability judgement 59/60 (98.3%) 38/60

(63.3%)

TROG (exclusive of single-word items) 65/68 (95.5%) 21/68

(30.9%)

Repetition of words 20/20 (100%) 9/20 (45.0%)

Repetition of non-words 17/20 (85.0%) 6/20

(30.0%)

Naming 36/60 (60.0%) 26/60 (43.3%)

29


Table 2: Overall movement rates and proportions specified by subjects and

movement categories. Data presented are totals over the sampled data (4.2 min

of speech).

S U B J E C T S

Movement variable MA PA AU

Number of PIMs 57 59 86

Total number of VCs 307 286 407

Number of VCs in PIMs 105 168 150

Number of VCs per PIM 1.8 2.8 1.7

The proportion of VCs-in-PIMs in total VCs 34.2% 58.7% 36.8%

The ratio of VCs-in-PIMs per word 0.201 0.235 0.374

VCs-in-PIMs per minute (rounded) 25 41 36

30


Table 3: Ideational gestures (IGs) and their timing in relation to speech.

S U B J E C T S

Movement variable MA PA AU

Number of PIMs 57 59 86

Number of IGs 42 28 66

The proportion of IGs in PIMs 73.7% 47.5% 76.8%

Conceptual IGs 30 23 29

Lexical IGs 10 3 32

Indeterminate IGs 2 2 5

The proportion of lexical in total IGs 23.7% 10.7% 48.8%

31


Acknowledgement

32

The research reported here was funded by the Israel Foundations Trustees (TheFord Fund).

Documents

CONCEPTUAL AND LEXICAL ASPECTS OF GESTURE: EVIDENCE FROM APHASIA