Embed Size (px)
Citation preview
Language and Perception
Ling 411 – 17
Perception: Starting view
Perception is a bottom-up process• From primary perceptual area upwards
E.g. primary auditory, for auditory perception Takes place in a single perceptual area
• E.g. auditory cortex for auditory perception Works by processing input to the sense organ
• E.g. auditory input for auditory perception
Each of these points is wrong!
The McGurk Effect
Acoustic syllable [ba] presented to subjects with visual presentation of articulatory gestures for [ga] Subjects typically heard [da] or [ga] “Evidence has accumulated that visual speech modifies
activity in the auditory cortex, even in the primary auditory cortex.”
Mikko Sams (2006)
http://www.youtube.com/watch?v=aFPtc8BVdJk
How does it work? (1)Visual input(2)Top-down processing
Perception – Refining a simple-minded view
1. It is not confined to a single perceptual modality• The McGurk effect
Auditory perception affected by visual input • Conceptual structure affects auditory perception
The influence of context on speech perception
Perception – Refining a simple-minded view
1. Not confined to a single perceptual modality• The McGurk effect
Visual input affects auditory perception• Conceptual structure affects auditory perception
2. Not just bottom-up• Top-down processing fills in unsensed details
3. Not even confined to posterior cortex• Can also use motor neurons
Experiment: left hand or right hand? Mirror neurons
Top-down processing in perception
TCUP
MADE OF GLASS
CERAMICSHORT
HAS HANDLE
Properties
Conceptual and perceptual information
Node for CUP in conceptual area for drinking vessels
Visual properties are in occipital and lower temporal areas
Bidirectional processing and inference
T
CUP
MADE OF GLASS
CERAMICSHORT
HANDLE
These connections are bidirectional
Pertinent neuroanatomical findings: Bidirectional Processing
An established fact of neuroanatomy:• A connection from point A to point B in
the cortex is generally accompanied by a connection from point B to point A
Separate fibers (axons): (1) A to B, (2) B to A
In short, cortico-cortical connections are generally bidirectional
Bidirectional processing: reciprocal links
excitatory
inhibitory
Bidirectional processing and inference
T
CUP
SHORT
HANDLE
Thought process: 1. The cardinal concept node is activated by a subset of its property nodes 2. Feed-backward processing activates other property nodes
Consequence: We “apprehend” properties that are not actually present in the sensory input
Bidirectional processing and inference
T
CUP
MADE OF GLASS
CERAMICSHORT
HANDLE
These connections are bidirectional
Separate fibers for the two directions; shown as one line in the notation
Cortical Structure and Inference:Perceiving things that are not in the input
T
Category
Properties
A
B F
E
Consequence: If A and B, then E and F
C D
Examples
Looks like a duck• Probably quacks
Ceramic, cup-shaped, handle• Probably holds coffee (without
breaking) Dark clouds, thunder
• It’s going to rain ATM
• Probably has money
Perception depends mainly on cortical structures already present before sensory input
“Perception is hallucination constrained by sensory data”
Shepherd
A terminological problem
We need to distinguish• Perception narrowly conceived
The basic process of recognition Single perceptual modality Bottom-up processing No motor involvement
• Perception broadly conceived Two different terms needed
• Recognition (a.k.a. ‘microperception’) Bottom-up process in a single perceptual modality
• Perception (the broad conception) (a.k.a. ‘macroperception’)
“Micro-perception” and “macro-perception”
Microperception and macroperception Microperception
• A.k.a. recognition• The local process of integrating features• Performed in one perceptual modality• Bottom-up
Macroperception• The overall process of perception• Uses multiple modalities• Uses top-down processing
Perception – Refining a simple-minded view
1. Not just bottom-up• Top-down processing fills in unsensed details
2. Not confined to a single perceptual modality• The McGurk effect
Visual input affects auditory perception• Conceptual structure affects auditory perception
3. Not even confined to posterior cortex• Can also use motor neurons
Experiment: left hand or right hand? Mirror neurons
Left hand or right hand?
Left hand or right hand?
Left hand or right hand?
Left hand or right hand?
Left or right hand?
Imaging experiment Subjects were shown pictures of one hand Asked to identify: left or right Functional imaging showed increased CBF in
hand area of motor cortex
Peter Fox, ca. 2000
Motor structures in perception
The left-hand vs. right-hand experiment ‘Mirror neurons’ in motor cortex Articulation as aid to phonological perception Articulation in reading Motor activity in listening to music Watching an athletic event
Mirror Neurons
NY Times: “One mystery remains: What makes them so smart?” (Jan. 10, 2006)
Answer: They are not smart in themselves• Their apparent smartness is a result of
their position: at top of a hierarchy• Compare:
The general of an army The head of a business
Similarly, high-level conceptual nodes• The “grandmother node”
Mirror Neurons
What makes mirror neurons appear to be special?• Ans.: They receive input from visual perception• The superior longitudinal fasciculus
Connects visual perception to motor areas How can a motor neuron receive perceptual input?
• Motor neurons are supposed to operate top-down• Answer: bidirectional processing
They also receive perceptual information• Bottom-up processing
Superior Longitudinal Fasciculus
From O. D. Creutzfeldt, Cortex Cerebri (1995)
Are some neurons “smarter” than others?
Claim: A grandmother node would have to be very smart • Identifies very complex object• Even in many varieties
Alternative: the head of a hierarchy• It is the hierarchy as a whole that has
those ‘smarts’• Similarly, mirror neurons
They get visual input since they are connected to visual areas
• Superior longitudinal fasciculus
Implications of hierarchical organization
Nodes at a high level in a hierarchy may give the appearance of being very “smart”
This appearance is a consequence of their position — at top of hierarchy
As the top node in a hierarchy, a node has the processing power of the whole hierarchy• Grandmother nodes• Mirror neurons• Compare:
The general of an army The head of a business organization
Multi-Modal Perception
1. Perception is not just bottom-up• Top-down processing fills in unsensed details
2. It is not confined to a single perceptual modality• The McGurk effect
Visual input affects auditory perception• Conceptual structure affects auditory perception
3. It is not even confined to posterior cortex• Can also use motor neurons
Motor activation in speech perception Mirror neurons
Perceptual structures in motor production
Perceptual structure is used in two ways1. Planning (e.g. visualizing while painting)2. Monitoring
Examples• Phonological recognition in speech production
Cf. Wernicke’s aphasia• Painting• Musical production• Baseball, soccer, tennis, etc.
The Influence of language on non-linguistic perception
As we have seen, non-speech input affects phonological perception
It is also the case that language affects non-linguistic perception• E.g., visual perception
Language and (Visual) Perception
Phonological Phonological Object Production Perception Categories
VisionLanguage
N.B.: These connections are bidirectional
Recent experiments of Kay et al.
Experiments at UC Berkeley• Color perception: do differences in color naming
across languages influence color perception? Main finding:
• Lateralized influence of language on perception• Response time faster for between-category
discrimination – especially for RVF presentation• A left hemisphere (RVF) phenomenon
green blue
Perception: A simple-minded view, revisited
Perception is a bottom-up process• From primary perceptual area upwards
E.g. primary auditory, for auditory perception Takes place in a single perceptual area
• E.g. auditory cortex for auditory perception Works by processing input to the sense organ
• E.g. auditory input for auditory perception
Each of these points in wrong
Another hypothesis of Whorf
Grammatical categories of a language influence the thinking of people who speak the language
Can we explain this too in terms of brain structure?
Mechanisms of operation
1. Entrenchment • Strengthening of connections
through repeated activation An automatic brain process Important in learning
2. Reverberation of activation 3. Priming4. Language as a major means of learning
conceptual and perceptual distinctions
Entrenchment and thinking: a mechanism
Connections become stronger with use• (entrenchment)
Grammatical categories make speakers constantly heed selected phenomena
Connections for phenomena which speakers must constantly heed.. • Will be repeatedly traversed• Therefore will get progressively stronger
Example: Grammatical gender
Does talking about inanimate objects as if they were masculine or feminine actually lead people to think of inanimate objects as having a gender?
Could the grammatical genders assigned to objects by a language influence people’s mental representation of objects?
Boroditsky (2003)
Plausibility of the possibility
Children learning to speak a language with grammatical gender may suppose that gender indicates a meaningful distinction between types of objects
Other grammatical distinctions do reflect actual perceptual differences: singular:plural
Children learning a language with gender
“For all they know, the grammatical genders assigned by their language are the true universal genders of objects.”
Boroditsky et al, 2003
Experiment: Gender and Associations(Boroditsky et al. 2002)
Subjects: speakers of Spanish or German• All were fluent also in English• English used as language of experiment
Task: Write down the 1st 3 adjectives that come to mind to describe each object• All the (24) objects have opposite gender
in German and Spanish Raters of adjectives: Native English speakers
Examples:
Key (masc in German, fem in Spanish)
• Adjectives used by German speakers: Hard, heavy, jagged, metal, serrated, useful
• Adjectives used by Spanish speakers: Golden, intricate, little, lovely, shiny, tiny
Bridge (fem in German, masc in spanish)
• Adjectives used by German speakers: Beautiful, elegant, fragile, peaceful, pretty
• Adjectives used by Spanish speakers: Big, dangerous, long, strong, sturdy,
towering
Results of the Experiment(Boroditsky et al. 2002)
Raters of adjectives were native English speakers
Result: Adjectives were rated as masculine or feminine in agreement with the gender in subject’s native language
end
