Embed Size (px)
Citation preview
Immersive Future Media TechnologiesSensory Experience
Christian Timmerer
Klagenfurt University (UNI-KLU) Faculty of Technical Sciences (TEWI)
Department of Information Technology (ITEC) Multimedia Communication (MMC)
http://research.timmerer.com http://blog.timmerer.com mailto:[email protected]
Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), and the ALICANTE project (FP7-ICT-248652).
Motivation
• Consumption of multimedia content may stimulate also other senses– Vision or audition– Olfaction, mechanoreception, equilibrioception, thermoception, …
• Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects
… giving her/him the sensation of being part of the particular media
➪ worthwhile, informative user experience
Outline
• Background / Introduction
– MPEG-V Media Context and Control– Sensory Effect Description Language (SEDL) and Sensory Effect
Vocabulary (SEV)– Software/Hardware components: SEVino, SESim, SEMP, and
amBX+SDK
• Improving the QoE through Sensory Effects Sensory ➪Experience
– A Brief Introduction to UME/QoE (UMA/QoS)– Results from Subjective Tests
• Conclusions and Future Work
MPEG-V: Media Context and Control
System Architecture
Pt. 1: Architecture
Pt. 3: Sensory Information
Pt. 4: Virtual World Object Characteristics
Pt. 2: Control Information
Pt. 6: Common Types and Tools
Pt. 7: Conformance and Reference Software
Pt. 5: Data Formats for Interaction Devices
http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V
Sensory Effect Description Language (SEDL)
• XML Schema-based language for describing sensory effects– Basic building blocks to describe, e.g., light, wind, fog, vibration, scent– MPEG-V Part 3, Sensory Information– Adopted MPEG-21 DIA tools for adding time information (synchronization)
• Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)– Extensibility: additional effects can be added easily w/o affecting SEDL– Flexibility: each application domain may define its own sensory effects
• Description conforming to SEDL :== Sensory Effect Metadata (SEM)– May be associated to any kind of multimedia content (e.g., movies, music, Web sites,
games)– Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate
mediation device
➪ Increase the experience of the user
➪ Worthwhile, informative user experience
Example
<sedl:GroupOfEffects si:pts="3240000" duration="100" fade="15" position="urn:mpeg:mpeg-v:01-SI-PositionCS-NS:center:*:front">
<sedl:Effect xsi:type="sev:WindType" intensity="0.0769"/>
<sedl:Effect xsi:type="sev:VibrationType" intensity="0.56"/>
<sedl:Effect xsi:type="sev:LightType" intensity="0.0000077"/>
</sedl:GroupOfEffects>
Quality of Experience
• Universal Multimedia Access (UMA)– Anywhere, anytime, any device + technically feasible– Main focus on devices and network connectivity issues QoS➪
• Universal Multimedia Experience (UME)– Take the user into account QoE➪
• Multimedia Adaptation and Quality Models/Metrics– Single modality (i.e., audio, image, or video only) or a simple combination of
two modalities (i.e., audio and video)• Triple user characterization model
– Sensorial, e.g., sharpness, brightness– Perceptual, e.g., what/where is the content– Emotional, e.g., feeling, sensation
• Ambient Intelligence– Add’l light effects are highly appreciated for both audio and visual content– Calls for a scientific framework to capture, measure, quantify, judge, and
explain the user experience
F. Pereira, “A triple user characterization model for video adaptation and quality of experience evaluation,” Proc. of the 7th Workshop on Multimedia Signal Processing, Shanghai, China, October 2005, pp. 1–4.
B. de Ruyter, E. Aarts. “Ambient intelligence: visualizing the future”, Proceedings of the Working Conference on Advanced Visual Interfaces, New York, NY, USA, 2004, pp. 203–208.E. Aarts, B. de Ruyter, “New research perspectives on Ambient Intelligence”, Journal of Ambient Intelligence and Smart Environments, IOS Press, vol. 1, no. 1, 2009, pp. 5–14.
Experiment: Goal & Setup
• Aim: demonstrate that sensory effects is a vital tool for enhancing the user experience depending on the actual genre
• Tools– Sensory Effect Media Player (SEMP)– Test sequences annotated with sensory effects: action (Rambo 4,
Babylon A.D.), news (ZIB Flash), documentary (Earth), commercials (Wo ist Klaus), and sports (Formula 1)
– Double Stimulus Impairment Scale (DSIS) also known as Degradation Category Rating (DCR)
• Five-level impairment scale ➪ new five-level enhancement scale
• Procedure– First, show reference sequence w/o sensory effects– Second, the same sequence enriched with sensory effects with a two
second break in between – Finally, subjects to rate the overall opinion of the audio/video resource
and sensory effect quality
Experiment: Results
Conclusions
• Sensory effects is a vital tool for enhancing the user experience leading to a unique, worthwhile Sensory Experience– Action, sports, and documentary genres benefit more
from these additional effects– Rambo 4 and Babylon A.D. are from the same genre,
the results differ slightly– Commercial genre can also profit from the additional
effects but not at the same level as documentary– News genre will not profit from these effects
• Interoperability through MPEG-V (ISO/IEC 23005)
Acknowledgments
• EC projects for funding this activity– NoE INTERMEDIA (NoE 038419)
• http://intermedia.miralab.ch/
– P2P-Next project (FP7-ICT-216217)• http://www.p2p-next.eu
– ALICANTE project (FP7-ICT-248652)• http://www.ict-alicante.eu
– COST ICT Action IC1003• QUALINET – European Network on Quality
of Experience in Multimedia Systems and Services
• Markus Waltl for implementing, preparing, conducting, evaluating almost all the experiments
• Benjamin Rainer for implementing the Firefox plug-in + WWW tests• Hermann Hellwagner for his advice and feedback• ISO/IEC MPEG and its participating members for their constructive
feedback during the standardization process
References
• Markus Waltl, Christian Timmerer, Hermann Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.
• C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, Proceedings of the NEM Summit 2009, Saint-Malo, France, September 28-30, 2009.
• M. Waltl, Enriching Multimedia with Sensory Effects, VDM Verlag Dr. Müller, February, 2010.
• M. Waltl, C. Timmerer and H. Hellwagner, “Increasing the User Experience of Multimedia Presentations with Sensory Effects”, Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, April 12-14, 2010.
• C. Timmerer, M. Waltl, and H. Hellwagner, “Are Sensory Effects Ready for the World Wide Web?”, Proceedings of the Workshop on Interoperable Social Multimedia Applications (WISMA 2010), Barcelona, Spain, May 19-20, 2010.
• M. Waltl, C. Timmerer, and H. Hellwagner, “Improving the Quality of Multimedia Experience through Sensory Effects”, Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX2010), Trondheim, Norway, June 21-23, 2010.
• M. Waltl, C. Raffelsberger, C. Timmerer, and H. Hellwagner, “Metadata-based Content Management and Sharing System for Improved User Experience”, Proc. of the 4th InterMedia Open Forum (IMOF 2010), Palma de Mallorca, Spain, September 1, 2010.
Thank you for your attention
... questions, comments, etc. are welcome …
Ass.-Prof. Dipl.-Ing. Dr. Christian TimmererKlagenfurt University, Department of Information Technology (ITEC)
Universitätsstrasse 65-67, A-9020 Klagenfurt, [email protected]
http://research.timmerer.com/Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
© Copyright: Christian Timmerer
