Standard approaches and a case study

1

Corso di Sistemi MultimedialiA.A. 2010-11

Prof. Antonio Picariello

Corso di Applicazioni TelematicheA.A. 2010-11

Prof. Simon Pietro Romano

Università degli Studi di Napoli Federico IIFacoltà di Ingegneria

9/6/2011

Outline Web Conferencing Introduction History and technologies

IETF standardization efforts

Milestones and ongoing works in the RAI area Centralized Conferencing, Media Control, Session Recording,

Via-browser conferencing, Telepresence Systems

From theory to practice: the Meetecho platform A fully-fledged standard conferencing system made in

unina

2

A “Web Conferencing” definition The term “Conference” can be used to describe any meeting

of people that “confer” about a certain topic

“Web Conferencing” is used to conduct live meetings or presentations over the Internet

3

Features VoIP (Voice over IP)

Live video

Text chat

Slide presentations

Whiteboard with annotation

Screen/desktop sharing

Application sharing

Recording

Polls and surveys

4

History Tele-Conferencing

Conference calls (Audio Tele-Conferencing)

Video conferences (Video Tele-Conferencing)

IP-Conferencing

Text Conferencing

Audio and Video Conferencing

Data Conferencing

5

Audio and Video ConferencingEvolution Analog Phone Lines (PSTN)

Conference calls Three-way calling

Conference bridges

Video Telephony

Digital Telephony (ISDN) ITU-T H.320 umbrella recommendation

Next-Generation Networks & VoIP technologies

6

VoIP protocols SIP – Session Initiation Protocol, RFC 3261

Call signalling

Session intitiation, managing and termination

Codec negotiation

SDP – Session Description Protocol, RFC 4566

Session media stream description

RTP – Real-time Transport Protocol, RFC3550

Media streams transfer

7

Text Conferencing Evolution Asynchronous Meetings

Posted text messages (not live)

Message/Bulletin Boards

Forums

Network Newsgroups/Mailing lists

Synchronous Meetings

Live text communication

Internet Relay Chat (IRC)

Web-based Chat (CGI/Java)

Instant Messaging (Skype/MSN/XMPP/etc.)

8

A B

C

A+B+CA+B

Typical scenarios From Point-to-Point Calls to Multipoint Calls

Three-way calling Coaching scenario

Lecture-mode Conferences Presentation Question & Answers session

Ad-hoc and Reserved Conferences Conference-aware/-unaware participants Manage conference/users/media/policies Sidebars/Whispers

9

Client B

Client CClient A

Media A Media C

Me

dia

B

Server

A

B

B

A

A+B

Issues Call signalling

Gateway functionality

Control and Management Tone detection (DTMF)

Dedicated protocols

Mixing and Transcoding Terminal capabilities

User media profiling Coaching scenario

Videoswitching

10

Client BClient A

SIP H.323

Server

Standardization Efforts Geographically dispersed businesses, the rising cost of energy, and the

development of NGN lead to the proliferation of proprietary conferencing solutions

No standardization for many years Lack of interoperability Platform dependency Security issues Cost Market segmentation

Standardization Bodies ITU (International Telecommunication Union) 3GPP (3rd Generation Partnership Project) IETF (Internet Engineering Task Force)

11

Standardization Efforts: ITU Established to standardize and regulate international radio and

telecommunications

International Standards referred to as “Recommendations”

ITU-T: Telecommunication Sector

G: Transmission Systems and Media G.71x (Audio compression, mu-law and a-law) G.72x (Audio compression, ADPCM)

H: Audiovisual and Multimedia Systems H.320 (PSTN/ISDN, Telephone Systems) H.323 (IP, Packet-based Communication Systems)

T: Terminals for Telematic Services T.120 (Data Sharing Protocols) T.140 (RTP Interactive Text)

12

Standardization Efforts: IETF Under the umbrella of the Internet Society

Develops and promotes Internet Standards

Deals in particular with standards of the TCP/IP suite

Organization

Working Groups (WG)

Internet Drafts

Requests for Comments (RFC)

“Rough consensus, running code”

We will focus on some RAI area activities...

13

IETF RAI area RAI – Real time Applications and Infrastructure

“Legacy” standards Standard VoIP protocols

SIPPING

XCON

MediaCtrl

Brand-new standardization activities SIPREC

RTCWEB

CLUE

14

SIPPING Working Group Session Initiation Protocol Proposal INvestiGation Documents the use of SIP for several applications related to

telephony and multimedia SIP Conferencing Models

15

Loosely-Coupled Conference Fully Distributed Multiparty Conference

Tightly-Coupled Conference

Sip ConferencingFramework

Main actors: - Participants- Focus- Mixer- Policy Server- Notification Server

XCON Working Group Centralized Conferencing (“X”CON)

A star topology on the signalling plane

Advanced conferencing features VoIP + video & data sharing

“Signalling-agnostic” Not only SIP

H.323, IAX, …

Defines : Conference data model and lifecycle A suite of client-server protocols to realize sophisticated

conferencing scenarios

16

Conference Object &Data Model

17

firstjoin

Template conference object

(blueprint)

Registered conference object

Activeconference object

cloningcreation

lastleave

<conference-info>

<sidebars-by-val>

<sidebars-by-ref>

<conference-description>

<host-info>

<floor-information>

<conference-state>

<users>delete

delete

Conference object example

18

<info:conference-info entity="8773158"><info:conference-description>

<info:display-text>My Full-Media Conf</info:display-text><info:conf-uris>

<info:entry><info:uri>xcon:8773158@meetecho.com</info:uri><xcon:conference-password>3903/0</xcon:conference-password>

</info:entry></info:conf-uris><info:available-media>

<info:entry label="audioLabel"><info:type>audio</info:type>

</info:entry><info:entry label="videoLabel"><info:type>video</info:type>

</info:entry></info:available-media>

</info:conference-description><info:conference-state>

<info:active>false</info:active></info:conference-state><info:users>

<info:user entity="13"><info:display-text>alex</info:display-text><info:endpoint entity="sip:alex@130.129.20.143:5060"/>

</info:user><info:user entity="14"><info:display-text>user134</info:display-text><info:endpoint entity="sip:user134@39.7.138.42:5080"/>

</info:user><xcon:join-handling>allow</xcon:join-handling>

</info:users><xcon:floor-information>

<xcon:floor-request-handling>confirm</xcon:floor-request-handling><xcon:conference-floor-policy><xcon:floor id="11"><xcon:media-label>audioLabel</xcon:media-label><xcon:max-floor-users>8</xcon:max-floor-users><xcon:moderator-id>13</xcon:moderator-id>

</xcon:floor><xcon:floor id="22">

<xcon:media-label>videoLabel</xcon:media-label></xcon:floor>

</xcon:conference-floor-policy></xcon:floor-information>

</info:conference-info>

Conference ID Title Conference state Password / PIN Allowed join modality Floor handling policy Participants info

User ID Nickname Signalling URI

Floor info Floor ID Associated media Moderator ID Max number of floor owners

XCON Protocols Signalling

SIP, H.323, IAX2, et al.

Floor Control BFCP

(Binary Floor Control Protocol)

Notification Xcon Event Package

Conference Control CCMP

(Centralized ConferencingManipulation Protocol)

University of Napoli highly active in this field

19

CCMP – Conference Control Conference CRUD Manipulation

Creation From a client-provided model or

by cloning system blueprints

Retrieval Update

User profiles, multimedia flow features (audio volume, video layout, …),...

Deletion

XML- based Carrying datamodel fragments

“Single verb HTTP + CCMP body” transport solution

20

BFCP – Floor Control Aimed at coordinating access to a set of shared resources

A “Floor” is a token, a temporary permission to access or manipulate a specific shared resource or set of resources

Standardized in RFC 4582 Identifiers (Conferences/Floors/Users) Floor Control Server Floor Control Participant Floor Chair

Negotiation of BFCP connections within SIP/SDP standardized in RFC 4583

Only existing implementation to date: COMICS/Ericsson

21

BFCP in action

22

1) Floor Request

2) Notify

3) Chair decision

4) Decision

5) Floor Granted/Denied

6) Notify

Chair Decision

1

2

3

4

5

6

Floor Request

Floor GrantedOr Denied

Decision Notify

Notify

MediaCtrl Working Group Defines an architecture in which an Application

Server can control a Media Server

A Media Server could be completely devoted toperform onerous online/offline operations on media flows: Media Processing

Mixing/Transcoding Playing/Recording Storing/Retrieving Detecting Tones (DTMF) Interactive Voice Response (IVR)/VoiceXML Text-to-Speech/Speech Recognition

RTP Streams Manipulation

Exploitable approach within conferencing frameworks

23

Client B

SIP SIP

Application

Server

Media

Server

SIP Control

Channel RTPRTP

SIPREC Working Group SIPREC - SIP-based RECording

Designing a protocol based on SIP-extension and a metadata model to deal withmultimedia session recording

A recorded session can play an important role in different scenarios E-learning, minutes taking, … Business analytics Media processing (online/offline) …

Some media processing actions may require separation between differentmedia streams E.g.:

Word-spotting, Speaker identification Emotion-detection …

24

SIPREC - Architecture Session Recording Client

(SRC) Logical entity able to

acquire the communicationsession to be recorded

Sends the recording media streams to the SRS

Session Recording Server (SRS) Records the received session

Recording Session (RS) ismade of: CS Multimedia Content CS Metadata

25

SIPREC - Requirements

“Active” recording The SRC proactively demands for registration No passive recording / sniffing / wiretapping

Session participants MUST be recording-aware Notification mechanisms are needed to inform participants

Start, Pause, Resume and Stop controls

Define a protocol between SRC and SRS to manage the recording session

26

SIPREC – Recording Metadata

Information that describes recorded media and the Communication Session (CS) to which they relate

They have to allow for:

Searching and filtering the archived session

Performing a “meaningful” playout

e.g., with correct synchronization between media streams

27

Recording Metadata Example<recording-metadata xmlns='...:siprec'>

<recording id=""> </recording>

<group id="" recording=""></group>

<session id=""group=""></session>

<participant id=""session=""></participant>

<stream id=""session=""></stream>

<extensiondata id="" parent="">

</extensiondata>

</recording-metadata>

Recording Metatada Elements Recording Session

Features: ID, recording type (selective, persistent), recording requestor (SRC or SRS), … One or mode associated CSG

Communication Session Group (CSG) Identified by a unique ID Group of one or more CS belonging to the same macro-session

Communication Session (CS) Features: ID, call termination reason, start time, end time, … Can involve one or more Participant elements and one or more Media Stream elements

Participant Features: name, AoR, participant role, …

Media Stream Features: start time, stop time, codec, SDP m-line reference, ..

Extension Data Envisioned the optional presenxe of additional data not specified as part of SIPREC

29

RTCWEB Working Group Real Time Communication on the World Wide Web

Goal: standardizing basic components enabling interactive rich communication using audio, video, collaboration, games, etc. between two peers' web-browsers Direct browser-to-browser communication

Nowadays several different methods exists, but they are not interoperable since they rely on incompatible browser plug-in platform

Collaboration with the W3C WEBRTC group : IETF architecture and requirements W3C Javascript API development

The idea: client-side applications, typically written in HTML and Javascript, interacting with web browser through a yet-to-be-defined API

30

RTCWEB IETF issues Identification and definition of network-related

aspects

Control protocols

Connection establishment and management

Connection-less transport

Selection of the most suitable encoders/decoders

Codec negotiation

New threats and security issues

Firewall and NAT traversal

Interoperation with legacy VoIP equipment, keeping the balance with practicality of browser deployment

31

CLUE Working Group

32

ControLling mUltiple streams for tElepresence

Telepresence systems provide high definition, high quality audio/video enabling a "being-there" /”immersive” experience

Use of multiple displays permitting life size imagereproduction using multiple cameras, encoders, decoders, microphones and loudspeakers

CLUE issues specifications for SIP-based conferencing systems

to enable communication of information about media streams necessary to make reasonable decisions about transmitting, selecting, and rendering media streams: Spatial relationships of cameras, displays, microphones, and

loudspeakers

Viewpoints - field of view/capture for camera/microphone/display/loudspeaker Senders and intermediate devices use them to understand how best to

compose streams for receivers

Receivers can know the characteristics of the received streams

Usage of the stream (presentation, document camera output, …)

Aspect ratio of cameras and displays

Sources selection receiver-driven

33

Meetecho Platform Web conferencing and on-line collaboration tool

Academic spin-off since Spring 2011 Developed by COMICS members and Unina students

Main features: Scalable and Performing Client-Server Solution IETF standards compliant (XCON, Media Ctrl) Open-source Java-based Desktop Client Mobile device access support

Android iPhone, iPad BlackBerry Symbian Windows Phone

Web browser access support (Meetecho Weblite)

34

Meetecho capabilitiesPSTN audio

Voice over IP audio

Live video

Audio & Video Moderation

Presentation delivery

Desktop/screen sharing

Text chat

Presence

File sharing

Whiteboard Sharing

Basic security

Remote Control

Session Recording

Polls & Surveys

Mobility support

Web access support

Meetecho in action

Audio/Video controls panel

Mirror On/Off

Pause Audio

Pause Video

Listen/Talk Volume

Selection of the desiredcontribution

Moderated Scenarios Realized by means of BFCP

The conference chair decides to grant or not audio and/or video floor to the requestor

The Meetecho server allows only granted users tocontribute their media within the conference media mix

Adaptive video layout depending on granted participantnumber

Slide sharing Slides are converted in background to a series of images made available on a

web server

All kind of presentation are supported Microsoft Office (ppt, pptx, doc, docx, xls, xlsx) OpenOffice (odp) Pdf Slides previously exported in image

formats as (gif, jpg, png, ecc.)

The presenter is the one who controlsslide progress

Slides can be sent to the whiteboardfor online collaborative modifications

Whiteboard Shared between conference participants

Allows to

Draw illustrative sketches

Modify slides

Insert clip-art

Save file modifications

Text Chat Based on XMPP (eXtensible Messaging and Presence

Protocol)

Allows instant messaging during conference sessions between conference participants

Features

Roster management

Presence info

Single file sharing

Conversation chronology

Polling Creating, answering and consulting real-time surveys

for conference participants

Desktop sharing Meetecho users can

Show to other participants their desktop activities

Render desktop control to other participants

Set visualization quality

Video mixing (1) Up to 16 parallel video streams in a single conference

Mixing strategy:

The mixer receives participant video contributions

Video streams are composed in a mosaico stream

The mixer sends to each participant the single mosaico stream and not each single video contribution That allows for

Constant load

Constant bandwidth consumption

Video mixing (2)

Server

MeetechoServer

Other tools overwhelm clients with as many flows as the contributing participants number…

…Meetecho mixes video streams and sends always a single flow to each users

Users

Users

User

User

Meetecho codecs

Supported audio codecs:

GSM

G.711 u-law

Supported video codecs:

H.323 Q-CIF (176 x 144 pixel)

H.323 CIF (352x288 pixel)

46

Managing Scalability Centralized case

A single focus for N users

Distibuted case Cascaded Conferencing

Load-balancing users between more interoperating focuses

Each focus is seen as a participant by the other focus

Orchestration of a set of XCON “clouds”

Experimental campaign showed performance benefits whit the increase of the participant number

47

Meetecho Session Recording Meetecho Session Metadata are encoded using SMIL

(Synchronized Multimedia Integration Language) Well known and established standard specification

Recordings are available in a standard fashion

Media timing – aware Enabling synchronized playout

Allowing media stream separation

A Recording Server is envisioned To receive all conference media streams and record them

To encode stream-related metadata info in the standard SMIL format

48

Server-side recording Client-side recording may be realized by means of ordinary desktop

recording softwares Available tools: Adobe Captivate, Wink, Istanbul, Byzanz,… Simple and fast Conferencing platfom independent

Client-side recording drawbacks: Only a single perspective is taken into account Recorded media “flattened” to a single, very large video Output files usually in proprietary formats

Flash, OGGTheora, …

Meetecho server-side recording approach uses standard formats is more suitable for multimedia session with several media and etherogeneous

devices involved

49

Recording audio & video Both Audio and video from

each participant are aggregated in the centralmixer the mixer can record and

store them locally or on a remote storage

Transcoding, adaptation and other operation may be carried out on the streams

The output is a single file containing both audio and video (AVI file with specific codecs)

Mixer UA-BUA-A RTPRTP

XCON AS

Recorder

MEDIACTRL

SIP SIP

RTP UA-A(Rx + Tx)

RTP UA-B(Rx + Tx)

Recording IM & events Chat contributions are

saved together withtimestamps and info on the employed IM protocol

51

Recorder UAC-BUAC-AB (IRC)

‘10:11:26 – Hey C’

A (XMPP)

UAC-C

C (MSRP)

‘10:11:30 – Hello’

‘10:11:24 – Hi!’

10:11:24 <User C> Hi!

10:11:26 <User A> Hey C

10:11:30 <User B> Hello

[…]

[…]

Textual event notificationmay be treated as well Slide changes

Join /leave events

Poll announcement

Recording slide presentations Presentations are considered as simple slideshows of

static images

Slide change events are triggered and notified via XMPP, registered as text

Slides are registered as a series of images togetherwith metadata info (associated events)

52

Recording whiteboarding

53

Whiteboard Server

UAC-BUAC-AB (XMPP)

10:10:56 – circle

A (XMPP)

UAC-C

C (XMPP)

10:12:30 – text

10:11:20 – line

10:10:56 <User A> circle

10:11:20 <User C> line

10:12:30 <User B> text

[…]

[…]

Putting It All Together: SMIL Standard XML-based markup language for describing

presentations involving heterogeneous media

SMIL metadata file

<head>

Generic metadata and layout composition details

<body>

References to media to be included (URL)

Layout regions info

Timing info

54

Media metadata SMIL allows for both standard and non-standard media to

be included in a presentation

Standard media streams have dedicated standard tags in the media part of the <body> <video> the A/V Meetecho recording AVI file

<textstream> IM, event notifications, converted in Real-Time Text docs

<img> slides

Non-standard media are included in the <ref> tag whiteboarding case

55

A Meetecho conf SMIL file

56

Playing out the recording (1)

57

Sessiondatabase

Web server

SMIL-enabled player

1. START

7. SHOW

2. get conf45.smil

video

text

slides

whiteboard

3. get audio/video

4. get RealText file

5. get slide images

6. get whiteboard packets

Playing out the recording (2) General approach:

1) Retrieval of the SMIL file (HTTP or out-of-bandmechanism)

2) SMIL file is passed to a compliant media player preparyingthe presentation layout

3) The player retrieves and reproduces each supported media

SMIL players examples : RealPlayer, Ambulant,… Standard stream support

Don’t play Meetecho whiteboarding…

Meetcho players!

58

Playing out the recordings (3) 3 options

1) Watch it online by means of a JNLP-based custom player

2) Watch it online withHTML5

3) Download the registration and watch it offline(jar file)

59

Session recordings available at:https:\\ietf.conf.meetecho.com

60