MPEG-V: Media Context and Control

MPEG-VMedia Context and Control

Christian Timmerer

Klagenfurt University (UNIKLU) Faculty of Technical Sciences (TEWI)Department of Information Technology (ITEC) Multimedia Communication (MMC)

http://research.timmerer.com http://blog.timmerer.com mailto:[email protected]

N107{82,83,85,86} – Text of ISO/IEC 23006-[1-4] CD

Christian Timmerer, Klagenfurt University, Austria 2

MPEG-V System Architecture

2009/07/03

Media context and control

Pt. 1: Architecture

Pt. 3: Sensory Information

Pt. 4: Avatar Information

Pt. 2: Control Information


Control Information

2009/07/03

Sensory Device Capabilities asext. of dia:TerminalCapability• unit, max/minIntensity, numOfLevels, delay, position• light (color, flash), heating, cooling, wind, vibration• scent, fog, water sprayer, color correction• kinestetic, tactile

User Sensory Preferences asext. of dia:UserCharacteristics• adaptability, max/minIntensity• light (color, flash), heating, cooling, wind, vibration• scent, fog, water sprayer, color correction• kinestetic, tactile

Fundamental Input to any Control

Device (aka Adaptation Engine)


Avatar Characteristics• Appearance

– Contains the high level description of the appearance and may refer a media containing the exact geometry and texture

• Haptics Properties– Contains the high level description of the haptics properties

• Animation– Contains the description of a set of animation sequences that the avatar is able to

perform and may refer to several medias containing the exact (geometric transformations) animation parameters

• Communication Skills– Contains a set of descriptors providing information on the different modalities an avatar

is able to communicate• Personality

– Contains a set of descriptors defining the personality of the avatar• Control

– Contains a set of descriptors defining possible place-holders for sensors on body skeleton and face feature points

2009/07/03


What is MPEG Sensory Information?

• UMA = Universal Multimedia Access– Access to content anywhere, anytime, any device == addressing QoS issues– Technical feasible thanks to standards ➙ MPEG-21 DIA

• UME = Universal Multimedia Experience– User-centric approach – takes into account the user– Quality of Multimedia Experience

• Sensorial, e.g., sharpness, brightness• Perceptual, e.g., what/where is the content• Emotional, e.g., feeling, sensation

• MPEG-V Sensory Information– Consumption of multimedia assets may stimulate also other senses, e.g., olfaction,

mechanoreception, or thermoception– Achieved by annotating the media resources with sensory effects

➙enhanced, worthwhile, and informative user experience, giving the user the sensation of being part of the actual media

2009/07/03

N9113

N10785


Concept of MPEG-V Sensory Information

2009/07/03


MPEG-V Sensory Information

System Architecture

2009/07/03

Scope of Standardization

Standardized representation device capabilities &➙

user preferences

Adaptation

Native representation ➙ standardized representation

Delivery

Authoring

Packaging


Committee Draft Structure

• Sensory Effect Description Language– Basic building blocks: GeneralInformation,

Declaration, Effect, GroupOfEffects, Parameter, ReferenceEffect

– Common attributes: XSI (timing), activate, duration, fade, alt, priority, intensity, adaptType, adaptRange

• Sensory Effect Vocabulary– Light (colored, flash), temperature, wind, vibration– Water sprayer, scent, fog, shadow, color correction– Kinesthetic, tactile

2009/07/03


Demo Setup

2009/07/03

Display

FanFan

Vibration Bar

Subwoofer

Wall-washer / Light

Light Light

Speaker Speaker


Demo Video with Effects

2009/07/03


Kinesthetic / Tactile Effects

2009/07/03

Viewers Television

Touch or exploration

Manipulation

Following movements

Feeling haptic effects

Haptic interaction

♬♩♪♫

Video

Audio

Haptic TextureForce

3D CG model

Text

abcdefg

Dynamics


Kinesthetic / Tactile Effects (cont’d)

2009/07/03

Active interactions

Passive interactions


MPEG-V SI References• N10782, Text of ISO/IEC 23006-1 CD• N10783, Text of ISO/IEC 23006-2 CD• N10785, Text of ISO/IEC 23006-3 CD• N10786, Text of ISO/IEC 23006-4 CD

• Ad-hoc Group– [email protected]– http://lists.uni-klu.ac.at/mailman/listinfo/metaverse

• Some selected publications– J. Cha, I. Oakley, Y.-S. Ho, Y. Kim, J. Ryu, “A Framework of Haptic Broadcasting”, IEEE Multimedia, in

print, Sept-Oct. 2009– J. Cha, S.-Y. Kim, Y.-S. Ho, J. Ryu, “3D Video Player System with Haptic Interaction based on Depth

Image-Based Representation”, IEEE Transactions on Consumer Electronics, Vol. 52, No.2, pp.477-484, May 2006

– M. Waltl, C. Timmerer, H. Hellwagner, A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.

2009/07/03

mailto:[email protected]

http://lists.uni-klu.ac.at/mailman/listinfo/metaverse


Thank you for your attention

... questions, comments, etc. are welcome …

Ass.-Prof. Dipl.-Ing. Dr. Christian TimmererKlagenfurt University, Department of Information Technology (ITEC)

Universitätsstrasse 65-67, A-9020 Klagenfurt, [email protected]

http://research.timmerer.com/Tel: +43/463/2700 3621 Fax: +43/463/2700 3699

© Copyright: Christian Timmerer

2009/07/03