Multisensory Interplay
Jon Driver and Toemme Noesselt
Sensory Research
Sensory Research
Behavioral Consequences of Multimodality
• Joint estimates of single propertyo Spatial Ventriloquismo Auditory Drivingo McGurk Effect
• Can modalities affect each other without creating a single unified percept?o Touch at a location can help
perception of coloro Sound-induced Illusory Flasho Orientation discrimination
improves with multiple beeps
Convergence Zones
• Superior Colliculuso inputs from somatosensory, auditory, and visual areaso Super or Sub-additive responses found for combination
Additivity in SC
• most likely with weak unisensory inputso Ceiling effect?o Neural limitations?
• Late onset in development
• Depends on multisensory cortex
Convergence Zones
Testing for Convergence
• Anatomical studies:o Direct connections between different sensory areas
• Single-cell studies:o Response to stimulation from different modalities
• Neuroimaging:o Large-scale responses based on BOLD signal
Influences on 'Sensory-Specific' Areas
• Growing body of research shows that sensory-specific areas might be an artifact of the studies done with them
• Examine studies that use:o fMRIo EEGo Invasive recording in animal models
fMRI
• Caveat:o fMRI has been shown to respond to attention and
imagery
• For example, speech may be imagined when viewing lip movements
fMRI analysis
• Inspired by Stein (SC), some look for sub-, super-additive responses
• Maybe linearity is normal, though, so some use max or mean criteria
• Difficult because of spatial resolution
Convergence in V1?
Amedi, Jacobsen, Hendler, Malach, and Zohary, 2002
ERP results example
• Tactile stimulation
• Visual cue
• ERP extracted
ERP studies
• visual N1 enhanced when tactile stimulation occurred at same location as a visual event
• visual P1 modified by task-irrelevant sound
• P1 modified by attend-visual relative to attend-tactile conditions
ERP studies
• ERPs show early multi-sensory effects (~30 ms)
• Poor localization
• Potential methodological confounds
Invasive Studies• Current-source densities
(CSD) reflect local PSPs
• Region of auditory association cortex
• Location and timing of stimulation consistent with auditory feed-forward, visual feed-back
Invasive Studies
• Posture may affect responses to auditory signals in A1
• Tactile stimuli modulate initial response to auditory signals in A1
Multisensory Interplay
• Examples of converging zones of multi-sensory input
• Examples of interplay: one modality affects another
• What frameworks does this evidence suggest?
Possible Frameworks
• A) All Multisensory
• B) Bimodal Brain Areas
• C) Critical Feedback Circuitry
All Multisensory
• Unlikely to be completely undifferentiated
• Even primary sensory areas responsive to multiple modalities
All Multisensory
• Thalamus might be source of multisensory interplay
• Tactile stimulation can affect first neural response in A1, hypothesized from thalamus
• Found in gerbils, hard to study in humans
All Multisensory
• Direct coritco-cortical influences
• Anatomical evidence: single synapse from AC-VC and AC-SSC, AC-OC
• However, not as many as to conventional Multi-sensory areas
• Role still unclear
All Multisensory
• Still overwhelmingly "sensory-specific"
Bimodal Brain Areas
• Less extreme version of account A
• Similar to current account, with more multi-sensory regions
• Parallel multi- and single-modality processing could explain early EEG modulation
Bimodal Brain Areas
• Different areas in auditory areas may be connected to distinct visual areas
• Bimodal interplay would be affected by transduction time, explaining BOLD response time differences
Feedback Circuitry
• Effects in primary areas might be feedback from convergence zones
• Evidence from effective connectivity in fMRI, tactile stimulation increases visual response
• Evidence from EEG source-localization: STS - VC
Feedback Circuitry
• Evidence from invasive recordings: late A1 stimulation from vision (speculation?)
• Feedback can be tested directly, but very little has been done
Remarks
• Perhaps the rival frameworks are all valid for certain situations
• Perhaps primary cortex responses in the blind and deaf can help tease out what the multisensory roles are
• New techniques will allow testing causal interplay