Understanding Action Verbs- Embodied Verbal Semantics Approach

Understanding Action Verbs- Embodied Verbal Semantics

Approach

Pavan Kumar SrungaramM.Phil Cognitive Science (09CCHL02)

2009-10Supervisor: Prof. Bapi Raju, S.

University of Hyderabad

Language in Two Different Perspectives• Chomskyan view: • Based on structures and rules. • Syntax is autonomous free of

meaning and general cognition• Human language is an abstract

formal system

Problems• In many cases it was observed that

background knowledge, concept, meaning have come into the rules governing syntax.

Language as a part of General Cognition• Lakoff, Fauconnier – Language not to

be studied in terms of structure/ Grammar but the way it is used

• Language and thought are embodied, concepts arise from our sensorimotor experience and the neural substrates that give rise to it.

• Metaphors We Live By !

Embodiment• Acknowledge the role of body and its

sensorimotor processes can and do play in cognition.

• Understanding cognition in the context of biological function to support the activities of the body.

• Cognition as real-time situated activity, inseparable from perception and action.

Behavioural Tests• Considered the experiments by

Bergen et al. (2010) and tried to replicate three of those tests in Telugu.

• In addition to these tests, we tried to verify whether the results produced in those experiments were modality specific

Behavioural Experiment• There are 4 tasks in this experiment• Each subject participates only in one

task• Task 1: Image-Verb Matching Task• Task 2: Verb-Image Matching Task• Task 3: Verb-Verb Matching Task• Task 4: Image-Verb (auditory)

Matching Task

Hypothesis• If verb comprehension requires the

activation of the motor areas related to the same effector with which the action is performed.

• Windhorst Hypothesis: Motor and perceptual systems display lateral inhibitions among neural structures responsible for related but incompatible functions

Materials and Methods• Computer Based Test • Two Stimuli presented at specific

times• Subjects are to judge as quickly as

possible whether the two stimuli match

• The response times of the subjects are recorded and analyzed

Materials and Methods• The stimuli used in the tasks are: a) Images: These contain a single

character performing some action b) Action Verbs written (or) typed both

in program compatible formats. c) Action Verbs (audio) • All these three are recorded and used

in program compatible formats.

Algorithm• First Stimuli: Presented for 1000

milliseconds• Inter stimulus interval: 500

milliseconds a) 450 milliseconds: Visual Mask

b) 50 milliseconds: Blank• Second Stimuli presentation: This is

until the subject makes his decision by pressing the designated key.

Presentation of Stimuli

Presentation of Stimuli

Task 1: Image-Verb Matching Task• Image is shown for a period of 1000

milliseconds

• 500 milliseconds inter stimulus interval

• This is followed by a verb in typed format

• Show images and verbs!

Results• The observation is in accordance

with the hypothesis.

• Subjects took longer time to reject the image-verb pairs of non-matching same effector condition compared to those in non-matching different effector condition.

Means Table

ConditionAverage RT in milliseconds Standard Deviation

Matching Condition 1281.34 444.4169

Non-Matching Same effector Condition 1502.599 431.3717

Non-Matching Different effector Condition 1394.467 386.6818

ANOVA: Analysis of Variance

RT

Sum of Squares df Mean Square F Sig.

Between Groups 1175122.403 2 587561.202 3.306 .040

Within Groups 2.506E7 141 177703.579

Total 2.623E7 143

Condition 1: Matching ConditionCondition 2: Non-Matching Same effector ConditionCondition 3: Non-Matching Different effector Condition

Results• The results obtained in the test

provide evidence that understanding action verbs, language users recruit neural resources that are normally used for performing an action. This also supports the claim that understanding language might require simulation of action.

Result of Image-Verb Matching Task

Test Statisticsa,b

RT

Chi-Square 7.939

df 2

Asymp. Sig. .019

a. Kruskal Wallis Test

b. Grouping Variable: Condition

Ranks

Condition N Mean Rank

RT 1.00 48 59.75

2.00 48 83.56

3.00 48 74.19

Total 144

Alternative explanation• An alternative explanation for the

observed effect is that seeing images might have led subjects to activate representations of other actions using the same effector and that the verb presented might be the actions related to those activated actions that might have resulted in delay in rejection

Task 2:Verb-Image Matching Task• Verb in typed format is shown for a

period of 1000 milliseconds


• This is followed by an image

Means Table

Condition

Average RT (in

milliseconds) Standard Deviation


Non-Matching Same

Effector Condition 1176.36 492.6629

Non-Matching Different



ANOVART

Sum of Squares df Mean Square F Sig.

Between Groups 612150.829 2 306075.415 1.384 .254

Within Groups 3.251E7 147 221140.118

Total 3.312E7 149

Result of Verb Image Matching Task

Ranks

Condition N

Mean Rank

RT 1.00 50 64.60

2.00 50 87.20

3.00 50 74.70

Total 150

Test Statisticsa,b

RT

Chi-Square 6.790

df 2

Asymp. Sig. .034



Results• As observed in task, the subjects

took longer time to respond when the verb and image share the same effector.

• Thus, to some extent it is evident that the results obtained in the test could not be due to the proposed alternative explanation for test 1 since the same effect is observed even when the order is reversed.

Task 3: Verb-Verb Matching Task• Verb in typed format is shown for a

period of 1000 milliseconds


• This is followed by a verb in typed format

Means Table

ConditionAverage RT (in

milliseconds)Standard Deviation


Non-Matching Same


Non-Matching Different


ANOVART Sum of

Squaresdf Mean Square F Sig.

Between

Groups795980.991 2 397990.496 3.351 .038

Within Groups 1.889E7 159 118782.609

Total 1.968E7 161


Result of Verb-Verb Matching Task

Ranks


RT 1.00 54 83.48

2.00 54 91.89

3.00 54 69.13

Total 162

Test Statisticsa,b

RT

Chi-Square 6.500

df 2

Asymp. Sig. .039



Results• When the images were replaced by

verbs and the task became a lexical-lexical matching task, and still the effect, as hypothesized, was observed

Question• In all the above tests, the

presentation of stimuli was through visual modality. This might raise a question whether the motor area activation is only due to presentation of stimuli in visual modality. To verify this, a study was conducted in which the images were displayed on the screen and the verbs are presented in auditory modality

Task 4: Image-Verb (Auditory) Matching Task• Image is shown for a period of 1000

milliseconds


• This is followed by a verb in auditory mode.

Means TableCondition Average RT (in

milliseconds)

Standard Deviation


Non-Matching Same 508.95 219.089

Non-Matching Different 440.08 144.3107

ANOVART Sum of

Squares df Mean Square F Sig.

Between

Groups128683.860 2 64341.930 2.213 .114

Within Groups 3575614.997 123 29070.041

Total 3704298.858 125


Result of Image-Verb (auditory) Matching Task

Ranks


RT

1.00 42 59.52

2.00 42 70.75

3.00 42 60.23

Total 126

Test Statisticsa,b

RT

Chi-Square 2.491

df 2

Asymp. Sig. .288



Result• It is evident from the table that RT in

the non-matching same effector condition is greater than those in non-matching different effector condition but it is found to be statistically insignificant.

.

SummaryCondition Task 1 Task 2 Task 3 Task 4Matching Condition 1281.34 1020.568 1353.48 438.85

Non-Matching same effector 1502.599

1176.36 1419.04 508.95

Non-Matching different effector 1394.467

1111.11 1247.92 440.08

Summary• We observe that irrespective of

statistical significance, the response times to reject pairs in non-matching same effector condition is larger than those in non-matching different effector condition.

SummaryTask TI: Picture-

Lexical Matching Task

T2: Lexical-Picture Matching Task

T3: Lexical-Lexical Matching Task

T4: Picture-Lexical (Auditory) Matching Task

Matching condition

7.45% 9.4% 11% 7.90%

Non matching same effector condition

20.90% 19.55% 18% 20.17%

Non matching different effector condition

6.12% 8.6% 7.33% 6.29%

Average error responses per Subject.

11.49% 9.71% 12.11% 11.45%

Summary• The higher rate of errors in the non-

matching same effector conditions in all the tasks is to be tested and given an explanation.

Conclusion• The results obtained in the various

tasks though may not completely support the claims of embodied semantics, can also not disprove its claims. This study shows that the notion of embodied verbal semantics is to be carefully investigated in other languages to verify the universality of the effect.

Thank you

Documents

Understanding Action Verbs- Embodied Verbal Semantics Approach