Upload
alexia-walsh
View
218
Download
1
Tags:
Embed Size (px)
Citation preview
The role of auditory-visual integration in object recognition
Clara Suied1, Nicolas Bonneel2 and Isabelle Viaud-Delmon1
1CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France2REVES / Inria Sophia-Antipolis, France
Acoustics’08
Research supported by the EU IST FP6 Open FET project CROSSMOD
Recognition of natural object
• Recognizing a natural object involves pooling information from various sensory modalities
• And to ignore information from competing objects
How do these multisensory information interact to form a unique object concept?
Object recognition
• To direct action to objects, spatial information needs to be encoded and this might interact with object perception– For recognition tasks with the spatial dimension not relevant to the task, conflicting results
(Gondan et al., 2005; Teder-Salerjarvi et al., 2005)
• Realistic object are of interest in the study of multisensory integration, since a given object can be identified through any of several single modalities– Little behavioural studies with realistic objects (e.g. Molholm et al., 2004 for an ERP study; Laurienti et al.,
2004 for linguistic-type stimuli)
•Identification task: go/no-go–When the target (telephone) is either heard or seen, press the button as fast as possible
–Withold response when distractor (frog) is presented alone
Main experiment: Object Recognition
Go conditions
A+40
RING RING
A+0V+
•Unimodal
RING RING
A+0V+ A+40V+
•Bimodal semantically congruent
CROAK CROAK RING RING
A+0V- A-0V+ A-40V+A+40V-
•Bimodal semantically incongruent
No-Go conditions
A-40
CROAK CROAK
A-0V-
•Unimodal
A-0V- A-40V-
•Bimodal semantically congruent
CROAK CROAK
Experimental questions
• Spatial alignment necessary for fast object recognition?
• Larger auditory-visual integration for realistic objects?
• Effect of distractors (semantic congruence) on performance?
Experimental questions
• Spatial alignment necessary for fast object recognition?
• Larger auditory-visual integration for realistic objects?
• Effect of distractors (semantic congruence) on performance?
2 (spatial alignment) x 4 (conditions) repeated-measures ANOVA
•Main effect of the spatial alignment (F1,19=17.68; p<0.0005)
•Main effect of the condition (F3,57=65.36; ε= 0.8; p<0.0001)
•But NO INTERACTION
the spatial effect is a Stimulus-Response Compatibility (Simon and Craft, 1970; Simon et al., 1981; Lu and Proctor, 1995)
Spatial alignment
Spatial alignment does not facilitate object recognition
Experimental questions
• Spatial alignment necessary for fast object recognition?
• Larger auditory-visual integration for realistic objects?
• Effect of distractors (semantic congruence) on performance?
• Computation of the effect size of the AV integration observed in the A+0V+ condition
(Cohen’s d; Cohen, 1988)
• Comparison with the size of AV integration previously observed in the literature
Size of the AV integration
2)]()),(min([
),min(
0
0
vAAV
VAAV
RTRTRT
RTRTRTd
Experimental questions
• Spatial alignment necessary for fast object recognition?
• Larger auditory-visual integration for realistic objects?
• Effect of distractors (semantic congruence) on performance?
• When the distractor is visual– No performance cost when processing an auditory target
• When the distractor is auditory– There is a performance cost when processing a visual target
It seems impossible to ignore an auditory distractor
Role of a distractor on object recognition
Conclusion
• Large bimodal integration effect– Size of the visual object, realism, 3D and large display, immersive
• No effect of spatial alignment on object recognition– Spatial alignment important for saccade generation or signal detection(Stein and Meredith, 1993; Hughes et al., 1994; Frens et al., 1995; Harrington and Peck, 1998)
– Object recognition is a function where spatial alignment is not essential
It could reflect the fact that this function probably involves brain regions containing neurons with broad spatial receptive fields
• A possible asymmetry in the attentional filtering of irrelevant auditory and visual information– Similar asymmetry for cueing effect in detection tasks (Schmitt et al., 2000)
– Alerting role of the auditory system?