N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Integrated Media Systems Center

Ulrich Neumann, DirectorCharles Lee Powell Professor of Engineering

Associate Professor, Computer Science

Isaac Maya, Ph.D., P.E. Director, Industry and Technology Transfer Programs

IMSC’s Immersive Technologies

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

NSF

Integrated Media Systems Center

a partnership among:National Science FoundationUniversity of Southern California

School of Engineering Annenberg Center for

CommunicationState of CaliforniaThe City of Los AngelesIndustry Partners: Computer Hardware and Software Entertainment Broadcasting Telecommunications Publishing Aerospace

Other Government Agencies:DARPA, NASA, JPL, ONR, US Army

$10 M operating budget

Downtown LA

IMSC Headquarters

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Faculty and Academia

A top-notch array of investigators – 14 of whom (~ 50%) were sought specifically because of IMSC

29 investigators working with 87 Phd, 26 MS, and 33 UG students

Investigators come primarily from EE and CS – others from Psychology, Industrial and System Eng., School of Cinema/Television, Annenberg School for Comm., School of Gerontology, Biomedical Engineering, and the Information Sciences Institute

Two IMSC investigators have PYI awards Seven IMSC faculty have CAREER awards (3 CAREER awards this year) Tomlinson Holman won a 2001 Technology Achievement Award from the Academy of

Motion Pictures Robert Sholtz won a MILCOM Achievement Award for UWB research (only 4th in 20 yrs)

IMSC faculty published 53 peer-reviewed journal articles and 205 peer-reviewed conference papers

articles appearing in print over a 12 month period in this past year (2001-2002)

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Education

Programs created by IMSC

171 students graduated with IMSC providing funding, classes, and research aspects of their education experience

81 with PHD, 77 with MS, and 13 with BS

IMSC created 3 MS programs enrolling 358 students unique among U.S. universities

IMSC created 2 UG minor programs enrolling 207 students

IMSC gave research fellowships to 32 UG students

Created more than 20 courses for programs – e.g.: Human Factors in Integrated Media Systems Integrated Media Systems and Architectures School of Engineering/Fine Arts (joint project course)

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Industry Program IMSC offers cost-effective research partnership at the 5-10 yr horizon

Pursue science and discovery of methods and algorithms – IP licenses Develop technology-impact and proof-of-concept before “start up” Prototype feasibility applications and testbeds Provide partners a view to potential opportunities and threats

Strong support through six years 28 members this past year (2001-2002)

Recognized 5 members for consistent partnership over 6 yr life of center HP, NCR, FXPAL, TRW, Lockheed Martin

Strong growth in international corporations and government programs Taiwan (III, ITRI) Korea (LG Electronics) Japan (FXPAL, Hitachi, NTT DoCoMo, Toshiba) Netherlands (Philips Research) Italy (ST Microelectronics)

Patents and tech transfer 46 patents filed over the life of center 24 patents licensed and 39 software licenses

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

IMSC Research Members

IIIITRI

Taiwan

Microsoft WA

See California Map

Collaborations with Government Agencies and Foundations:

US ArmyDARPANASA Toyota Foundation

NCR OH

Eastman Kodak NY

DRS TechnologiesPrentice Hall PTR

NJ

NTT DoCoMo Japan

NavTech IL

Intel OR

Time Domain AL

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Silicon Valley

Hollywood

Los Angeles

Silicon Valley

•ConceptLabs•F-X Palo Alto Laboratory•Geometrix•Hewlett-Packard•InterVideo•Lockheed Martin•Sun Microsystems

USCIMSC

Los Angeles/San Diego

• AboveNet Communications• Boeing• HRL• JPL• Los Angeles Times• NMUA• Panoram Technologies• TMH Corporation• TRW

IMSC Research Members

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

IMSC Research Program -- ImmersipresenceMotivated by Moore’s law: exponential performance growth...

enables advances towards the future of the internet, grid computing, nano-technology, ...

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

People are often the ultimate purpose behind silicon systems

In pursuit of an experience, or the performance of a task, or understanding of information

Integrated Media System (IMS) mediate the gap between silicon and humans

An extension of the human senses, actuators, or cognitive process

Our goal is to advance the science and engineering of “well designed” Integrated Media Systems that exploit human information processing capabilities - providing effectiveness that we associate with “immersipresense”.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

To better understand immersipresence and help set

specific goals, we develop visions of how IMS

technologies will affect people and their activities

We examine four domains for IMS impacts in the future

These four vision scenarios drive the center’s program of six research and engineering areas

UWB console captures 3D model and haptics scan

PDA provides spatial array of participants and sonic

translation of haptic trace

Immersive workstation coordinates video, avatars, graphics, audio, and haptics to facilitate problem resolution

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Immersive Audio

Facial Gesture Analysis and Animation

Robust Vision Analysis

Immersi-data Content Extraction & Analysis

Theory of Perceptual and Cognitive Pleasure

User State Sensing and Perceptual User Interfaces

Virtual Environments for Performance Testing and Training

Multimedia Networks, Transmission, and Comm-unication

Ultra Wideband Wireless

Remote Media Immersion

Software Architecture for Immersi-presence

HAPTICS – devices and translations

Immersive on-demand streaming of audio/video experiences - fusion of cinema and internet

Virtual/video peer-to-peer telepresence experiences for

multiple participants

Full body immersion role-playing game experiences with realistic or synthetic characters

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Immersive Audio

Speech Recognition and Synthesis

Facial Gesture Analysis and Animation

Robust Vision Analysis

Digital Geometry Processing

Immersive Media Real-Time Storage & Retrieval

Emotional Expressions

User-State Sensing and Perceptual User Interfaces

Multimedia Networks, Transmission, and Comm-unication

Ultra Wideband Wireless

Remote Media Immersion

Software Architecture for Immersi-presence

Remote Media Immersion - fusion of internet and cinema

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Theory of Perceptual and Cognitive Pleasure

Emotional Expressions

User State Sensing and Perceptual User Interfaces

Virtual Environments for Performance Testing and Training

Information Integration

Immersi-data Content Extraction & Analysis

Customized Querying & Rendering

Immersive Media Real-Time Storage & Retrieval

Remote Media Immersion

Software Architecture for Immersi-presence

RMI - fusion of internet and cinema

BioSIGHT - authoring and assessment

I-NEWS - story models and user preferences

HAPTICS - devices and translations

Immersive audio

Speech Recognition and Synthesis

Facial Gesture Analysis and Animation

Robust Vision Analysis

Digital Geometry Processing

Multimedia Networks, Transmission, and Comm-unication

Ultra Wideband

Wireless

Compression

IMSC research program

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Research Highlights

immersive audio multichannel and HRTF approaches - holistic DSP approach

streaming servers distributed and scalable architecture

UWB wireless leadership in technology, FCC regulation, and commercialization

perception & cognitive modeling theoretical foundations for understanding benefits of media immersion

computer vision computational framework for grouping based on tensor voting

graphics & animation

3D DSP mesh processing, facial expression analysis and animation, hair modeling

virtual reality -- chair of VR2003 in LA applications to psychology (ADD diagnosis), haptics applications, and user studies

IMSC has produced ground breaking results and fundamental research in:

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

IMSC Immersipresence Vision: Applications in Entertainment

Animation File Goes Here:PC – panoramic_animation.aviMAC - panoramic_animation.mov

360º Camera

USC Homecoming Game at the Los Angeles Coliseum

Panoramic Visual Immersion: Malibu Hills

QuickTime™ and aCinepak decompressor

are needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

IMSC Immersipresence Vision: Medical Applications

Animation File Goes Here:

PC – 200_0003.AVI

Evaluation of ADHD in Children

QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Video FilesMac: face_1.movPC: avatars_doug.avi

Animation and Modelingof People for Immersipresence

3D Visual Environment

Avatar simulates live video

Avatar: Digital Cloning

QuickTime™ and aCinepak decompressor

are needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Animation and Modeling:Expression Cloning

Real Expression can be “cloned” to any model

QuickTime™ and aNone decompressor

are needed to see this picture.

QuickTime™ and aNone decompressor

are needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Video FilesMac: face_1.movPC: avatars_doug.avi

Animation and Modeling:Expression Analysis

Real Expression can be “cloned” and analyzed

QuickTime™ and aNone decompressor

are needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Digital Geometry Processing: 3D Modeling and Elasticity

Animation File Goes Here:PC – mat_1.aviMAC - mat_1.mov

Animation File Goes Here:PC – mat_2.aviMAC - mat_2.mov

Animation File Goes Here:PC – mat_3.aviMAC - mat_3.mov

Next wave of visual data: Better understanding and manipulation of geometric data

QuickTime™ and aCinepak decompressor

are needed to see this picture.

QuickTime™ and aCinepak decompressor

are needed to see this picture.

QuickTime™ and aCinepak decompressor

are needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Mixed Reality and Visualization

Augmented Reality for Space Flight (NASA funded)

Develop AR authoring tools for video-based training

Anthony Majoros, Human Factors group, The Boeing Corp.

4D Battlefield Visualization (MURI funded)

Develop fusion of video, images, and 3D models Avideh Zakhor (Berkeley), Suresh Lodha (UC

Santa Cruz), Bill Ribarsky (Georgia Tech), Pramod Varshney (Syracuse)

Wide Area AR Tracking (ONR funded)

Novel sensors and fusion for tracking position/orientation

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Non-negative Negative

(From the same person and dialog)

Recognizing Emotions in Speech

Natural Human-computer interaction: need for machines that can recognize “what, who and how” Call center applications, Teleconferencing, Tutoring by

machines, Entertainment Algorithms (ASRU’01, ICME’02, ICSLP’02)

Support Vector Machine, Linear discriminant and k-NN classifier for acoustic information

Novel information-theoretic measure of emotionalsalience in language (word sequences)

Acoustic and linguistic information fusion Performance evaluated on real call center data

rather than feigned data from actors

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Video FilesMac: audio_10.2_channel.movPC: audio_10.2_channel.avi

Immersive Audio 10.2

3D Virtual audio field at the location of the ears at all times

Immersive Audio Environment

QuickTime™ and aCinepak decompressor

are needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Haptics Research

Force control for haptics rendering Collision detection Compression of haptic data Haptic playback and analysis Application Research Project: Haptic Museum

CyberGrasp PHANToM (3D Mouse)

Movie File:PC: haptic_2.aviMAC: haptic_2.mov

Movie File:PC: haptic_1.aviMAC: haptic_1.mov

QuickTime™ and aCinepak decompressor

are needed to see this picture.

QuickTime™ and aCinepak decompressor

are needed to see this picture.

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Haptics Experiment

Virtual “touching” through the internet Virtual “handshaking” Distributed Haptic World

INTERNET

Mutual Touch

CyberGrasp PHANToM

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Media CommunicationsHigh Bandwidth Networks: synchronization, latency, bandwidthHigh Bandwidth Networks: synchronization, latency, bandwidth

Wireless SystemsWireless Systems

Optical CommunicationOptical Communication

CompressionCompression

Photonic NetworksPhotonic Networks

QoS Management of IP NetworksQoS Management of IP Networks

Encryption and Network SecurityEncryption and Network Security

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Managing ImmersidataHaptic Application

Control

Immersive Audio Text3D Model

Avatar

Video

Immersidata

Real-time acquisition and playback Efficient query and analysis Integration of disparate information sources

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

With the aim to With the aim to improve:improve:

Effectiveness Efficiency Enjoyment Safety

Human Factors Human Computer Interaction General Experimental Psychology Usability Engineering Communication User-Centered Design Cognitive Science Ethnography

“User Centered Sciences” Research

Integrates Multiple Methodologies:

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Systems Engineering

RMI testbed for streaming media immersionShowcase for IMSC research and integrationMay 2002 demonstration (NY Times)

Flow Scheduling Framework of the Media Immersion Environment - open source release

BioSIGHT authoring tools and web-based educationTesting and assessment in K-12 classrooms

Similarly, unique systems engineering and integration are hallmarks of IMSC

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

RMI A fusion of internet browsing with a theater-like

immersive experience HD Video at up to 45 Mbits/sec 10.2 ch Immersive audio (16 Mbits/sec)

Steaming over the Internet on-demand to a mouse click (20 min prog)

RMI demonstrates IMSC innovations in several critical technical areas:

Immersed in a college football game

Doctors assisting in a remote procedure

Business people negotiating like they

are in the same room

Students visiting an aquarium a thousand

miles away

Streaming media

servers

Immersive audio

capture and

rendering

Network protocols for error

correction Synchronization

Applications

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

RMI Challenges Immersive, high-quality video and

audio acquisition and rendering High Definition video 1080i and

720p (~40 Mb/s) 10.2 channels of uncompressed

audio (12 Mb/s)

Storage and transmission of media streams across networks Streaming media

servers/clients Error correction via selective

retransmission Synchronization between

streams A/V, A/A, V/V

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

RMI Architecture

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Storage, Streaming & Rendering Distributed

Server Storage Scheduling Scalability

Clients Multi-stream

Synchronized playback

Transmission Robust VBR

Flow control

Requirements:

A streaming platform that can scale and handle synchronized, high-bandwidth streams

Focus:

End-to-end streaming architecture

[ACM Computer’02]

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Scalability: Multi-Node, Multi-Disk Data and control network traffic can be routed with

different logical topologies Yima-1: single data path

(high inter-node traffic) Yima-2: multiple data paths

(low inter-node traffic)

Yima-2Yima-1

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Robust Stream Delivery

Variable bit rate (VBR) media encoders allocate more bits to complex scenes and less bits to simple ones

Smoothing of VBR media traffic has the following quality benefits: Better resource utilization (less bursty) More streams with the same network capacity

Multi-Threshold Flow Control(MTFC) algorithm objectives: Online operation Content independence Minimizing feedback

control signaling Rate smoothing

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Premier Demonstration May 9, 2002

Cross-country streaming from ISI-East in Virginia

NY Times coverage in “Circuits” section

NBC-TV and KTLA http://imsc.usc.edu/rmi/ http://www.east.isi.edu/NGI-

S/

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Future Impact of RMI High-quality audio/video experience streamed on-

demand over the internet (live or recorded) Everyone is a content producer, distributor, theater

Currently only broadcast, cable, theater, or satellite operators can distribute content…

high entry cost and regulation barriers RMI technology removes distribution barriers

CEO NVidia: “the end of TV as we know it” LIVE concerts, sporting events, alternative news

coverage, … RECORDED movie archives, news archives,

education programs, … “More than TV” experience – powerful human motivation

Playback quality and options are flexible – retain all the capabilities of computing on the internet

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Distributed Immersive Performance Next generation of Remote Media Immersion (RMI) Vision – interactive peer-to-peer streaming

Create seamless immersive environment for distributed high-quality audio/video interaction

Impact – widespread telepresence Distributed performances, meetings, social

gatherings, education, Technical barriers

Precise timing and synchronization of streams Low latency, error correction – high quality-of-service

network protocols

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Summary IMSC’s vision of Immersipresence and Integrated Media

Systems aims for fundamental breakthroughs in technology, industry partnerships and education

IMSC is carrying out Highly cross-disciplinary cutting edge research Aggressive industry collaboration and technology

transfer Unique educational experience and teamwork

skills for students

IMSC is eager to expand its Collaborations

Acknowledgments

National Science Foundation Engineering Research Center program

http://www.eng.nsf.gov/eec/erc.htm

IMSC Industry partners.....

More information about IMSC

http/:imsc.usc.edu

IMSC PARTNERS Airborne 1 Corporation Compression Science CorporationConceptLabsEastman KodakF-X Palo Alto LaboratoryGeometrixHewlett-PackardHRLIBMInformixInstitute for Information IndustryIntelInterVideoJPLLittonLockheed MartinLos Angeles TimesLucent Technologies/Bell LabsNCRPanoram TechnologiesPhilips Multimedia CenterPrentice Hall PTRRaytheonSTMicroelectronicsSun MicrosystemsTMH CorporationTRW