The Brain and Multimodal Communication

Elisabeth AhlsénUniversity of Gothenburg

SSKKII Interdisciplinary Center & Department of Linguistics

Nationella Forskarskolan i Kognitionsvetenskap ht 2008

Contents1. Communication on different levels of

awareness and control - corresponding brain functions

2. What models cover - view of cognition and communication

3. Multimodal communication - some controversies

4. Examples of current research trends5. Brain damage - cognition and

communication

1. Communication on different levels of awareness and control -

corresponding brain functions

Communicative Body Movements:1)Facial gestures (nose, eyebrows, cheek, forehead, chin etc)2)Head movements3)Gaze direction, mutual gaze4)Lip movement5)Hand and arm movements6)Leg and foot movements7)Body posture8)Distance btw communicators9) Spatial orientation10) Touch11) Non-linguistic sounds

Homunculus

Types of representation

3 ways of carrying, conveying and sharing information

• Index• Icon• Symbol

Variation in awareness and intentionality

Bodily communication often morespontaneous, unaware, unintended (index, icon) than words (symbols).

Three levels of intentionality:

index icon symbol

indicate x

display x

signal x

indicate -> display -> signal

conventional signal -> automatic

Dimensions of content in communication

Words BodyFactual information

Emotions and Attitudes

Physiological states

Identity

Social relations

Communication ManagementInteractive communication managementOwn communication management

Interactive view on embodied communicationTwo communicators form one dynamic system by

establishing stabilizing communication links

Max Feedback + Max OutlookFeedback

Feedback employs all of these levels, too.

Communication - Cognition

Memory (WM, LTM)

Attention (sustained attention)

Central executive

…

Right hemisphere lesions

• Left neglect, spatial disorder

• Prosody disorders

• Lexical-semantic disorders

• Problems with emotion information

• Discourse disorder (complexity, ToM, metaphor, humor, irony, etc)

Primitive brain - Limbic system - Neocortex

James-Lange vs Cannon-Bard

Formation of emotions

AmygdalaMediation and control of major affective activities like friendship, love and affection, on the expression of mood and, mainly, on fear, rage and aggression

The center for identification of danger. Fundamental for self

preservation. Humans with marked lesions of the amygdala, loose the affective meaning of the perception of an outside information, like the sight of a well known person. The subject knows, exactly, who the person is, but is not capable to decide whether he likes or dislikes him (or her).

Hippocampus

Formation of long-term memory.When both hippocampi (right and left) are destroyed, nothing can be retained in the memory. The subject quickly forgets any recently received message. The intact hippocampus allows the animal to compare the conditions of a present threat with similar past experiences, thus enabling it to choose the best option.

Fornix & parahippocampal gyrus

Connecting pathways

Thalamus

Lesion or stimulation of the medial dorsal and anterior nuclei of the

thalamus are associated with changes in emotional reactivity. The medial dorsal nucleus makes connections with cortical zones of the

pre-frontal area and with the hypothalamus. The anterior nuclei connect with the mamillary bodies, and through them, via fornix, with the hippocampus and the cingulate gyrus.

HypothalamusLesions of the hypothalamic nuclei interfere with several vegetative functions and some of the so-called motivated behaviors.Specifically, its lateral parts seem to be involved with pleasure and rage, while the median part is like to be involved with aversion, displeasure and a tendency to uncontrollable and loud laughing. However, in general terms, the hypothalamus has more to do with the expression (symptomatic manifestations) of emotions

Cingulate gyrus(gyrus cinguli)

Its frontal part coordinates smells and sights with pleasant memories of previous emotions. This region also participates in the emotional reaction to pain and in the regulation of aggressive behaviour

Brain stem

Responsible for the "emotional reactions", (indeed, they are just reflex answers) of inferior vertebratesRemains active, not only as alerting mechanisms, vital for survival, but in the maintenance of the sleep-awake cycle.

Septum

Pleasant sensations, mainly those related to sexual experiencesAnterior to the thalamus

Prefrontal areaIts intense bi-directional connections with thalamus, amygdala and other subcortical structures, account for the important role it plays in the genesis and, specially, in the expression of affective states.

When the pre-frontal cortex suffers a lesion, the subject looses his sense of social responsibility as well as the capacity for concentration and abstraction. In some cases, although consciousness and some cognitive functions, like speech, remain intact, the subject can no longer solve problems, even the most elementary ones.

2. What models cover - view of cognition and communication

A. Localist-associationist ”classical” modelLanguage - cortical centers + subcortical white

fibres-Broca’s area - production-Wernicke’s area - perception-comprehension-Connections-Concept centerGeschwind

B. Dynamic localization of function

A. R. Luria

Functional systems of many subfunctions in different areas, not always language specific, (e.g. rapid alternations, inertia)

Dynamic systems, can be restored-changed

3 blocks: brain stem, posterior PTO, anterior

C. Hierarchical / evolution based models(”holistic”)

J.H. Jackson- J. Brown: Microgenesis”from deep to surface” in phylogenesis,

ontogenesis and microgenesis”Focus on role of subcortical, older structures,

emotion etc.Automatic vs. Controlled behavioronLevels of function for language, emotion,

perception

3. Multimodal communication - brain and cognition - some controversies

• Localism - Associationism - Holism• Modularity - Interaction• Symbol manipulation - Distributed activation

/representation• Linear models - Interactive Activation

Models (IAM,ANN)• Language/speech and gesture intertwined or

separate/compementary - related how?• Definition/delineation of language / communication

system• Different types of body movements - how are they related

and how do they relate to spoken language?

4. Examples of current research trends

- Embodiment - many aspects, gesture one of them, role of body for development of language another

- ”Growth point” (McNeill)- Cultural influence vs universality of gesture +

relation to speech- Building embodied artificial agents- Studying coordination-communication on lower

levels of awareness and control- Neuroimaging? - Action-Perception link - localization

5. Brain damage & multimodal communication

From language/words sentences to communication - multimodal

Widening of the concept of aphasia (cf. language) - controversy

From left hemisphere cortex to any language-communication disorder, including, RH, subcortical-cerebellar, distributed, e.g. Alzheimer etc., TBI

New approaches needed and substantial research

New models needed

Next episode…

On

evolution,

mirror neurons,

the role of imitation and pantomime

and possible revisions of theories about communication disorders…

Assignment

Choose one of the controversies mentioned above and write one page argumenting for each of the two sides or suggesting two different views and approaches to answering one of the questions.

Literature:

Arbib M (2005). From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics. Brain and Behavioral Sciences, 28, 105–24.

Ahlsén, E. 2008. Neurological disorders of embodied communication. In.I. Wachsmuth, G. Knoblich & M. Lenzen (eds.) Embodied Communication in Humans and Machines. Oxford: Oxford University. Press.