View
219
Download
0
Category
Tags:
Preview:
Citation preview
MACHINE-MEDIATEDCOMMUNICATION
3D environments,(sensors, projectors: sight, sound, touch, other sensory modes)
information capture, display
personalagents multimodal
fusion
storage
media conversion, multilingual translation
information representation
computationmanagement
architectures, protocols
agentsdB’s
libraries
storageWeb pages
grid-based computation
Other Sites
HumanInterface
GlobalNetwork
CommunicationMediator
security
security
Multimodal Human-Machine Interface
Force Feedback Glove
Gaze Tracker
Microphone
Array
Multimodal Human-Machine Interface
Real-world trial with NJ National Guard
Portable Interactive Command Console (PICC)
Steerable microphone arrayelement
Stereo face tracking cameras
Gaze tracker
Source locator microphone
Loudspeakers
Light pen
Flatpaneldisplay
Sensors
HQ/VEHICLE
Robotic Vehicles Emergency Responder in the Field
Internet
User interface for interaction and collaboration with robots and humans
NSF Equipment Grant EIA#98-18313 Center for Advanced Information Processing, Rutgers University, Piscataway, NJ 08854.
PI: J.L. Flanagan, co-PIs: J. Wilder, I. Marsic, M. Krane
“Move this to there.”
FRAMEWORK FOR MULTIMODAL LANGUAGE
SENSORY SIGNAL
IDENTIFY ACTION
ESTIMATE MEANING
GENERATE RESPONSE
FORM SENTENCE
CREATE “WORDS”
CLASSIFY SEGMENTS
SEGMENT CONTINUA
EXTRACT FEATURES
Transduce intelligence
Options library
Semantic rules; parse
sentence
Multimodal language
generation
Syntax rules; word n-grams
Lexicon; grammar;
symbol n-grams
Symbol library
Source – receiver
constraints
Transform domain
FUSION OF MODALITIES
Recommended