Embed Size (px)
DESCRIPTION
On cross-modal sensory streams. Invited Panel Presentation at SIGGRAPH '98, Orlando, Florida. The v OIC e by Peter B.L. Meijer. Philips Research Laboratories [email protected]. The v OIC e: “Seeing with your Ears”. Basic facts: - PowerPoint PPT Presentation
Citation preview
© 1998 Peter B.L. Meijer
On cross-modal sensory streams...
The vThe vOICOICeeby by
Peter B.L. Meijer Peter B.L. Meijer
The vThe vOICOICeeby by
Peter B.L. Meijer Peter B.L. Meijer
Philips Research LaboratoriesPhilips Research [email protected]@philips.com
Invited Panel Presentation at SIGGRAPH '98, Orlando, Florida
© 1998 Peter B.L. Meijer
The vOICe: “Seeing with your Ears”
Basic facts:Basic facts:
• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear
• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations
Basic facts:Basic facts:
• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear
• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations
© 1998 Peter B.L. Meijer
The vOICe: “Seeing with your Ears”
Basic facts:Basic facts:
• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear
• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations
Basic facts:Basic facts:
• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear
• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations
VideoVideo
AudioAudio
MentalMentalVideoVideo
VideoVideo
AudioAudio
MentalMentalVideoVideo
The vOICeThe vOICeThe vOICeThe vOICe
Human brainHuman brainHuman brainHuman brain
So translate video to audio!?So translate video to audio!?So translate video to audio!?So translate video to audio!?
© 1998 Peter B.L. Meijer
The vOICe implementations
HardwareHardwareprototypeprototypeHardwareHardwareprototypeprototype
Java appletJava appletJava appletJava applet
WindowsWindows95/98/NT95/98/NT
WindowsWindows95/98/NT95/98/NT
http://www.seeingwithsound.com/http://www.seeingwithsound.com/
© 1998 Peter B.L. Meijer
The vOICe image-to-sound mapping
SumSum of ofsinusoidssinusoids
SumSum of ofsinusoidssinusoids
Fre
quen
cyF
requ
ency
Time
Time
AmplitudeAmplitude
© 1998 Peter B.L. Meijer
The vOICe image-to-sound mappingF
requ
enc
Fre
que
ncyy
TimeTime
Spectrographic Spectrographic synthesissynthesis
Spectrographic Spectrographic synthesissynthesis
© 1998 Peter B.L. Meijer
Could this really work?Could this really work?
CheckCheck:: map sounds back to images map sounds back to images
Could this really work?Could this really work?
CheckCheck:: map sounds back to images map sounds back to images
The vOICe image-to-sound mapping
Yes!Yes!(at(at leastleast technicallytechnically))
Yes!Yes!(at(at leastleast technicallytechnically))
TheoryTheoryTheoryTheory
© 1998 Peter B.L. Meijer
The vOICe image reconstruction
““Soundscape”Soundscape”““Soundscape”Soundscape”
The vThe vOICOICeeThe vThe vOICOICee
SpectrogramSpectrogramSpectrogramSpectrogram
© 1998 Peter B.L. Meijer
The vOICe image reconstruction
““SoundscapeSoundscape””““SoundscapeSoundscape””
© 1998 Peter B.L. Meijer
The vOICe and human speech
PronouncingPronouncingone imageone image
per secondper second
PronouncingPronouncingone imageone image
per secondper second
© 1998 Peter B.L. Meijer
The vOICe mental imagery?
RepresentationRepresentationmay dramaticallymay dramaticallyaffect affect recognitionrecognition!!
RepresentationRepresentationmay dramaticallymay dramaticallyaffect affect recognitionrecognition!!
© 1998 Peter B.L. Meijer
The vOICe limitations
Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty
• Physiology / Perception:Physiology / Perception:
just-noticeable difference (JND)just-noticeable difference (JND)
critical bands (human cochlea)critical bands (human cochlea)
Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty
• Physiology / Perception:Physiology / Perception:
just-noticeable difference (JND)just-noticeable difference (JND)
critical bands (human cochlea)critical bands (human cochlea)
© 1998 Peter B.L. Meijer
The vOICe limitations
Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty
• Physiology / Perception:Physiology / Perception:
just-noticeable difference (JND)just-noticeable difference (JND)
critical bands (human cochlea)critical bands (human cochlea)
? auditory streaming/segregationauditory streaming/segregation
? neural pathways & bandwidthneural pathways & bandwidth
? neural processing & plasticityneural processing & plasticity
Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty
• Physiology / Perception:Physiology / Perception:
just-noticeable difference (JND)just-noticeable difference (JND)
critical bands (human cochlea)critical bands (human cochlea)
? auditory streaming/segregationauditory streaming/segregation
? neural pathways & bandwidthneural pathways & bandwidth
? neural processing & plasticityneural processing & plasticity
© 1998 Peter B.L. Meijer
The vOICe limitations
Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty
• Physiology / Perception:Physiology / Perception:
just-noticeable difference (JND)just-noticeable difference (JND)
critical bands (human cochlea)critical bands (human cochlea)
? auditory streaming/segregationauditory streaming/segregation
? neural pathways & bandwidthneural pathways & bandwidth
? neural processing & plasticityneural processing & plasticity
Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty
• Physiology / Perception:Physiology / Perception:
just-noticeable difference (JND)just-noticeable difference (JND)
critical bands (human cochlea)critical bands (human cochlea)
? auditory streaming/segregationauditory streaming/segregation
? neural pathways & bandwidthneural pathways & bandwidth
? neural processing & plasticityneural processing & plasticity
• Psychology / Education:Psychology / Education:
? sounds (too) unpleasantsounds (too) unpleasant
? minimum required resultsminimum required results
? acceptable training effortacceptable training effort
? motivation for learningmotivation for learning
? training programstraining programs
• Psychology / Education:Psychology / Education:
? sounds (too) unpleasantsounds (too) unpleasant
? minimum required resultsminimum required results
? acceptable training effortacceptable training effort
? motivation for learningmotivation for learning
? training programstraining programs
© 1998 Peter B.L. Meijer
The vThe vOICOICe approache approach needs more researchneeds more research
is cheap technologyis cheap technology
is non-invasiveis non-invasive
The vThe vOICOICe approache approach needs more researchneeds more research
is cheap technologyis cheap technology
is non-invasiveis non-invasive
Conclusions
© 1998 Peter B.L. Meijer
The vThe vOICOICe approache approach needs more researchneeds more research
is cheap technologyis cheap technology
is non-invasiveis non-invasive
is available is available Now!Now!
The vThe vOICOICe approache approach needs more researchneeds more research
is cheap technologyis cheap technology
is non-invasiveis non-invasive
is available is available Now!Now!
Conclusions
http://www.http://www.seeingwithsoundseeingwithsound.com/.com/
![Cross-Modal Interpretation of Multi-Modal Sensor Streams ... · sensor signals for robot perception [3]. They enhance robot perception by exploiting interaction as an additional source](https://img.dokumen.tips/doc/110x75/5ea343d752780b0b226c18a0/cross-modal-interpretation-of-multi-modal-sensor-streams-sensor-signals-for.jpg)
