Research Group 'Multimedia Communication' Presentation (March 2015)

Univ.-Prof. Hermann Hellwagner

Assoc.Prof. Christian Timmerer

Institute of Information Technology (ITEC)

Faculty of Technical Sciences (TEWI)

Alpen-Adria-Universität Klagenfurt (AAU)

Austria

Research Group Multimedia Communication (MMC)

1

Research Group

Multimedia Communication (MMC)


2

Alpen-Adria-Universität Klagenfurt (AAU)

> 10,000 students, > 800 employees

(> 500 scientific staff members, 70 full

professors)

Studies: cultural sciences, economics,

psychology, mathematics, computer

science, information technology, …


3

Faculty of Technical Sciences (TEWI)Application EngineeringIntelligent Systems and Business InformaticsSystemsicherheitInformation and Communication SystemsInteractive SystemsSoftware EngineeringDistributed Multimedia Systems

Multimedia Communication

Mobile SystemsPervasive ComputingEmbedded SystemsSmart GridsTransportation InformaticsSensory and ActuatorsControl and Mechatronic Systems

Applied AnalysisDiscrete Mathematics and OptimizationEducation and Philosophy of Mathematics

Biometrics and Clinical StatisticsSpatial and Financial StatisticsBiometrics and Educational Statistics

Full Professor

• Dr. Hermann Hellwagner

Associate Professor

• Dr. Christian Timmerer

Secretary

• Margit Pertl, BA

University Assistants

• DI Daniel Posch

• DI Florian Bacher

System Engineer

• Rudolf Messner


4

Research Group Staff

Project Assistants

• DI Christian Kreuzberger

• DI Daniela Pohl

• DI Christian Raffelsberger

• DI Benjamin Rainer

• Sandra Bergmann

Multimedia Communication and QoS/QoE Provisioning

Adaptive Video Streaming

Dynamic Adaptive Streaming over HTTP (DASH)

Use of Scalable Video Coding (SVC) Technology

Multimedia Content Distribution in Information-Centric Networks (ICN)

Quality of Experience Beyond Audio-Visual Perception

Sensory Experience

Standardization within ISO/IEC MPEG

Multimedia in Disaster Management


5

Research Areas

European Commission (IST FP6 /FP7):

• DANAE (STREP; 01/2004-06/2006): „Dynamic and distributed Adaptation of scalable multimedia

coNtent in a context-Aware Environment“

• ENTHRONE II (IP; 09/2006-11/2008): „End-to-End QoS through Integrated Management of

Content, Networks and Terminals“

• INTERMEDIA (NoE; 10/2006-09/2010): „Interactive Media with Personal Networked Devices“

• P2P-Next (IP; 01/2008-12/2011): „Next Generation Peer-to-Peer Content Delivery Platform“

• ALICANTE (IP; 03/2010-08/2013): „Media Ecosystem Deployment Through Ubiquitous Content-

Aware Network Environments“

• BRIDGE (IP; 04/2011-03/2015): „Bridging Resources and Agencies in Large-Scale Emergency

Management“

• SOCIAL SENSOR (IP; 10/2011-10/2014): „Sensing User Generated Input for Improved Media

Discovery and Experience“

Industry (partial funding):

• MPEG-21 standardization: in cooperation with Siemens CT IC 2, Munich

• SVC adaptation: in cooperation with ST Microelectronics, Milan


6

Research Projects and Funding

COST (European Cooperation in Science and Technology):

QUALINET (ICT Action IC1003; 11/2010 – 11/2014): „European Network on Quality

of Experience in Multimedia Systems and Services“

Austrian Science Fund (FWF) – Basic Research:

• Quality Adaptive Video Caching and Transport (QAVCT)

• Adaptive Streaming of Secure Scalable Wavelet-based Video (ASSSV)

• A Context-Adaptive Content Ecosystem Under Uncertainty (CONCERT)

Austrian Research Promotion Agency (FFG) and CELTIC:

• Scalable Video Based Provisioning of TV and VoD Services over the Internet

(SCALIPTV)

Klagenfurt University:

• Independent PhD Theses


7

Research Projects and Funding

www.p2p-next.org

General goal: development of the open-source, BitTorrent-based,

next-generation P2P system NextShare

Approach: research, prototyping, and trials

in Klagenfurt and around the world

Specific goal of MMC:

streaming of scalable media in P2P mode


8

Example of Previous Work: P2P-NextPiece-Picking for Layered Content in BitTorrent-based P2P Networks

http://www.p2p-next.org/

When streaming layered videos in a BitTorrent-based P2P network,

the piece-picking algorithm needs to decide which piece to download

at which point in time.

The main goal is to provide the best

possible quality with the available

bandwidth while ensuring continuous

playback and minimizing changes

in quality.


9

P2P-NextPiece-Picking for Layered Content in BitTorrent-based P2P Networks


10

P2P-NextSliding Window and Utility Calculation

𝑼𝒊 𝒋 𝒌 =𝐝𝐣 × 𝐝𝐩𝐢 𝐣 𝐤

(𝐢𝐩𝐢 − 𝐢𝐩𝐤)𝛂

ipi: start of the ith time slot in intervals

ipk: kth decision point in intervals

lj: layer of the piece

dj: distortion reduction importance of the piece

dpijk: probability to receive the useful piece in time

: urgency weighting

The support for layered content and the piece-picking algorithm for

layered content was integrated into the NextShare P2P system.

The performance of the newly developed piece-picking algorithm was

evaluated using the Omnet++/Oversim simulation framework as well

as by using real system tests.

Compared to distributing single-layer content, the support for layered

content can significantly increase the performance when realistic peer

churn rates and/or less than optimal network conditions are applied.

Compared to previously published piece-picking algorithms for

layered content, the newly developed piece-picking algorithm

provides a very good performance over all tested scenarios.


11

P2P-NextResults

www.bridgeproject.eu

General goal: development of technical solutions for support of

emergency agencies that help increase safety of EU citizens in case

of large-scale emergencies

Technical solutions are developed with a focus on:

Interoperability: interoperability between heterogeneous (ad-hoc)

networks, different data formats and existing contextual information

Situational awareness: common operational picture through

developing intelligent user interfaces, data collecting, filtering, and

information sharing mechanisms

Collaboration: inter-agency workflow creation and composition


12

Examples of Current Work: BRIDGEBridging Resources and Agencies in Large-Scale Emergency

Management

http://www.bridgeproject.eu/

Technical solution to improve situational awareness in emergency

management through sub-event detection from multimedia data (especially,

social media data)

Sub-events are specific hotspots originating in the context of an emergency

(disaster, crisis) that are of importance for planning the response activities

(e.g., flooding in a specific district, collapsed buildings in another area)

Sub-events are recognized via identifying multimedia items describing the

same incident

Sub-event detection algorithms are examined, created, and evaluated

covering offline (retrospective) and online (just-in-time) detection algorithms

Important techniques: machine learning (i.e., feature selection, clustering, and

classification), active learning, natural language processing


13

BRIDGESub-Event Detection from/with Multimedia Data

Multimedia (Detection) Framework that allows to include multimedia data from

different sources (e.g., on-site through an Android app, specific social media

platforms, etc.) in the sub-event detection process

Transmission of identified sub-events to other BRIDGE related sub-systems

to support a common

operational picture

Training support for

emergency management

through the development

of a simulation tool to

simulate social media

usage during an exercise


14

BRIDGESub-Event Detection from/with Multimedia Data

Multimedia (Detection) Framework

Processing

(Aggregation/Detection & Filtering)

BRIDGESub-Systems

Social Media Platforms

Android-App

Simulation Tool

Result Monitoring

Result Visualization

Identifying & transmitting important information

BRIDGEMiddleware

Simulation scenario examples

Technical solution to improve communication in ad-hoc networks for

emergency response operations that are prone to disruptions

Development of hybrid routing approaches that combine end-to-end and

delay-tolerant routing in order to mitigate effects of network disruptions and

provide communication in partitioned networks

Providing disruption-tolerant communication services to other BRIDGE

systems such as electronic triage

or S.O.S. smartphone app

Simulation-based evaluations

of hybrid approach in

realistic emergency

response networking

scenarios


15

BRIDGECommunication in Disrupted Networks

Use-case example

Multimedia communication in disrupted networks

Videos from incident site are important to improve situational awareness and

provide a common operational picture

State-of-the art multimedia delivery protocols cannot cope with disruptions

Development of a multimedia

delivery system that allows to

exchange multimedia data

between the incident site and

the incident command center

in partitioned ad-hoc networks


16

BRIDGEMultimedia Communication in Disrupted Networks

www.concert-project.org

International basic research project, cooperation partners:

University College London, Dept. of Electronic and Electrical Engineering

University of Surrey, Centre of Communication Systems Research

École Polytechnique Fédérale de Lausanne, Signal Processing Lab.

General goal: to develop a content ecosystem with all relevant

“players“ which will be able to perform intelligent content and network

adaptation in highly dynamic conditions under uncertainty

Basis: Information-Centric Networks (ICN) with Content- and Context-

aware Routers (CCR)


17

CONCERTA Context-Adaptive Content Ecosystem Under Uncertainty

http://www.concert-project.org/

CONCERTA Context-Adaptive Content Ecosystem Under Uncertainty


18

ContentServer

MultimediaContent

CCR – Content- and Context-aware Router

Cached and adapted multimedia content

ICN is a revolutionary network architecture which changes the current

host-centric paradigm to a content-centric paradigm

Specific ICN under investigation: Named Data Network (PARC, UCLA)

General goal: improve the effectiveness of content dissemination and

provide the best possible Quality of Experience (QoE) for consumers

Scenarios: commercial content distribution; user-generated content;

multi-view streaming

NDN‘s security model enables network-inherent caching, but prohibits

in-network content processing/adaptation in the first place

In-network adaptation is being integrated based on layered encodings


19

CONCERTEffective Content Dissemination and In-Network Adaptation in ICN

dash.itec.aau.at (Dynamic Adaptive Streaming over HTTP – DASH)

Goal: efficient, adaptive multimedia streaming over HTTP/TCP

Approach: research and development, prototyping, measurements,

and standardization of DASH concepts


20

Adaptive Media Streaming

http://dash.itec.aau.at/

selab.itec.aau.at

What comes after 3D Video? Sensory Effects

Goal: to improve the user‘s QoE by means of sensory effects

Approach: annotate videos with additional effects (wind, light,

explosion, etc.) which are rendered on specific devices (fan, vibration

chair, etc.), subjective tests, QoE models


21

Sensory Experience Lab

… giving her/him the sensation of being part of the particular media➪ worthwhile, informative user experience

http://selab.itec.aau.at/

www.socialsensor.eu

Snychronized playback of Internet video streams with concurrent real-

time communication channels

Inter-Destination Media Synchronization (IDMS)


22

Social TV

Live Sport Events- Olympic Games- NFL Superbowl- FIFA Worldcup- …

SatelliteBroadcast

(Semi-)Professional&

AmateurContent Acquisition

Consumer

Consumer

Consumer

Heterogeneous Broadband Networks

(wired, wireless)

Geographically distributed consumers accessing the same content while communicating among them.

Syn

chro

niz

ed

Pla

you

t

Online Community

http://www.socialsensor.eu/

www.qualinet.eu (COST Action QUALINET)

Factors impacting Quality of Experience


23

Quality of Experience (QoE)

Quality of Experience

(QoE)

DeviceNetwork

Content Format

EnvironmentContent

User Expectation

TaskApplication

Technical Factors

Social andPsychological

Factors

User

Context

http://www.qualinet.eu/

www.bitmovin.com

Spin-off from MMC and research projects ALICANTE, SocialSensor

Founders: Stefan Lederer, Christopher Müller, Christian Timmerer

Investors: Speed Invest GmbH, Constantia Industries AG


24

bitmovin

http://www.bitmovin.com/

MPEG-4 ISO Base File Format: timed metadata

MPEG-21 Multimedia Framework (2002-2007)

Editor of the standard

Major contributions to Digital Item Adaptation

Various contributions to Vision, Technologies and Strategy, Digital Item Declaration,

File Format, Reference Software, Conformance, Event Reporting, Binary Format

MPEG-V Media Context and Control (2007-2010)

Editor of the standard

Development of Sensory Effect Description Language (SEDL); reference impl.

Multimedia Extensible Middleware: MXM (2008-2010)

Co-editor of the standard

Contributions related to adaptation of multimedia content according to the usage

environment context


25

MPEG Standardization Activities

MPEG Media Transport (2010)

Chair of the AhG and the evaluation of the responses to the call for proposals

MPEG-M Advanced Media Platform Technologies (2010-2012)

Co-editor of the standard

Various contributions in the area of content adaptation

Dynamic Adaptive Streaming over HTTP (2010 – now)

Chair of the AhG and the evaluation of the responses to the call for proposals

Contributions related to the composition of media presentations, quality metrics, and

conformance / reference implementation (co-editor of the standard)

Head of the Austrian Delegation


26

Recent MPEG Standardization Activities

Pohl D., Bouchachia A.: Information Propagation in Social Networks during Crises: A Structural

Framework. In: D. Krol, D. Fay, B. Gabrys (eds.): Propagation Phenomena in Real World Networks.

Springer 2015 (to appear).

Timmerer C., Waltl M., Rainer B., Murray N.: Sensory Experience: Quality of Experience Beyond

Audio-Visual. In: S. Möller, A. Raake (eds.): Quality of Experience: Advanced Concepts, Applications

and Methods. Springer 2014.

Hellwagner H.: The Interplay of Technology Development and Media Convergence: Examples. In: S.

Diel, M. Karmasin (eds.): Media and Convergence Management. Springer 2013.

Hellwagner H., Kofler I., Eberhard M., Kuschnig R., Ransburg M., Sablatschan M.: Scalable Video

Coding: Techniques and Applications for Adaptive Streaming. In: C. Zhu, Y. Li, X. Niu (eds.): Streaming

Media Architectures, Techniques and Applications: Recent Advances. IGI Global 2011.

Grafl M., Timmerer C., Hellwagner H., Negru D., Borcoci E., Renzi D., Mevel A., Chernilov A.: Scalable

Video Coding in Content-Aware Networks: Research Challenges and Open Issues. In: N. Blefari-

Melazzi, G. Bianchi, L. Salgarelli (eds.): Trustworthy Internet. Springer 2011.

Hellwagner H., Timmerer C.: MPEG-21 Applications and Use Cases. In: M.C. Angelides, H. Agius

(eds.): The Handbook of MPEG Applications: Standards in Practice. Wiley 2010.


27

Relevant Recent PublicationsBooks and Book Chapters

Pohl D., Bouchachia A., Hellwagner H.: Online Indexing and Clustering of Social Media Data for

Emergency Management. Neurocomputing, 2015 (to appear).

Rainer B., Timmerer C.: A Generic Utility Model Representing the Quality of Sensory Experience. ACM

Transactions on Multimedia Computing, Communications, and Applications (TOMM), 11(1s):14:1-

14:17, Sept. 2014.

Ghinea G., Timmerer C., Lin W., Gulliver S.: Mulsemedia: State of the Art, Perspectives, and

Challenges. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM),

11(1s):17:1-17:23, Sept. 2014.

Timmerer C., Rainer B.: The Social Multimedia Experience. IEEE Computer, 47(3):67-69, March 2014.

Pohl D., Bouchachia A., Hellwagner H.: Social Media for Crisis Management: Clustering Approaches

for Sub-Event Detection. Multimedia Tools and Applications, Dec. 2013.

Grafl M., Timmerer C., Hellwagner H., Xilouris G., Gardikis G., Renzi D., Battista S,. Borcoci E., Negru

D.: Scalable Media Coding Enabling Content-Aware Networking. IEEE MultiMedia, 20(2):30-41, 2013.

Waltl M., Rainer B., Timmerer C., Hellwagner H.: An End-to-End Tool Chain for Sensory Experience

based on MPEG-V. Signal Processing: Image Communication, 28(2):136-150, Feb. 2013.

Hellwagner H., Hofbauer H., Kuschnig R., Stütz T., Uhl, A.: Secure Transport and Adaptation of MC-

EZBC Video Utilizing H.264-based Transport Protocols. Signal Processing: Image Communication,

27(2):192-207, Feb. 2011.

Lopez F., Jannach D., Martínez J., Timmerer C., Garciá N., Hellwagner H.: Bounded Non-deterministic

Planning for Multimedia Adaptation. Journal of Applied Intelligence, 36(1):29-60, Jan. 2010.


28

Relevant Recent PublicationsInternational Journals

Raffelsberger C., Hellwagner H.: A Multimedia Delivery System for Delay-/Disruption-Tolerant

Networks. In: Proc. 13th IEEE Int‘l. Conf. on Pervasive Computing and Communications Workshops

(PerCom Workshops) 2015 (to appear).

Kreuzberger C., Posch D., Hellwagner H.: A Scalable Video Coding Dataset and Toolchain for

Dynamic Adaptive Streaming over HTTP. In: Proc. 6th ACM Int‘l. Conf. on Multimedia Systems

(MMSys) 2015.

Rainer B., Timmerer C.: Self-Organized Inter-Destination Multimedia Synchronization For Adaptive

Media Streaming. In: Proc. 22st ACM Int‘l. Conf. on Multimedia (MM) 2014.

Rainer B., Timmerer C.: A Quality of Experience Model for Adaptive Media Playout. In: Proc. 6th Int‘l.

Workshop on Quality of Multimedia Experience (QoMEX) 2014.

Raffelsberger C., Hellwagner H.: Combined Mobile Ad-Hoc and Delay/Disruption-Tolerant Routing. In:

Proc. 13th Int‘l. Conf. on Ad-hoc, Mobile, and Wireless Networks (ADHOC-NOW) 2014.

Posch D., Kreuzberger C., Rainer B., Hellwagner H.: Client Starvation: A Shortcoming of Client-driven

Adaptive Streaming in Named Data Networking. In: Proc. 1st ACM Int‘l. Conf. on Information-Centric

Networking (ICN) 2014.

Pohl D., Bouchachia A., Hellwagner, H.: Online Processing of Social Media Data for Emergency

Management, In: Proc. 12th Int‘l. Conf. on Machine Learning and Applications (ICMLA) 2013.

Lederer S., Mueller C., Rainer B., Timmerer C., Hellwagner H.: An Experimental Analysis of Dynamic

Adaptive Streaming over HTTP in Content Centric Networks. In: Proc. IEEE Int‘l. Conf. on Multimedia

and Expo (ICME) 2013.


29

Relevant Recent PublicationsInternational Conferences