TSIN02 - Internetworking · TSIN02 - Internetworking 8 Technical Issues There are two fundamental issues when designing a real-time communications system: 1) Synchronization. Sender

TSIN02 - Internetworking

© 2004 Image Coding Group, Linköpings Universitet

Lecture 7: Real-time Streaming

Literature:– Fouruzan ch. 25.3-25.7, Multimedia– RFC3550 RTP, RTCP: 1-6.2, 6.4, 7-7.1– RFC2327 SDP (optional extra material)– RFC2326 RTSP (optional extra material)

Ver 2.00 Jonas Svanberg


2

Lecture 7: Real-time StreamingGoals:

● Understand “real-time”● Be aware of the problems of

– Synchronization

– Jitter

and their solutions● Understand the RTP architecture of senders, receivers,

mixers and translators● Understand some uses for the sister protocol RTCP.● Understand how synchronization is achieved in RTP● Know about NTP, SDP and RTSP


3

Lecture 7: Real-time Streaming

Outline:

● Real-time communication in general

● Synchronization

● RTP

– Terminology– Header structure– Transport– Interleaving

● RTCP (Real-time Control Protocol)

● RTP Synchronization Model

● RTP Software● Network Time Protocol● SDP – Session

Description Protocol● RTSP – Real-time

Streaming Protocol


4

Real-time definition

What is “real-time”?● Real-time communication: No noticeable delay● Real-time application: An application able to have a

sustained processing of a temporal data stream– Without having queues building up– With reasonable sustained quality level

But - there is of course always delay to some extent! ● Different communication scenarios may have different

sensitiveness to delay.


5

Real-time Communication

Real-time communication involving humans typically fall into at least one of the following two categories:● Dialog situation - when a user has some kind of a interaction

with other user(s) or machines – Telephony– TV query shows where you may phone in.

Some dialogs like controlling a VCR is not so delay sensitive...● Moment sharing – the feeling of sharing a unique event with

other people. Notion of being among the select few who first know what's going on.

– TV sports broadcast

Order of tenthsof a second

Order of secondsusually ok


6

Real-time Scenario

Coder DecoderBuffer

02:05:00124Transmitter Reference Clock

02:06:16126Receiver Reference Clock

Jitter makes thepackets arrive irregularly

Network induces delayand jitter

Frames createdregularly atfrequency f

Buffer smoothsout the effectof jitter


7

Typical Design Decisions

When having real-time communication we usually● Skip retransmission

– The time it takes to send ACK:s/NACK:s upstream and wait for the lost packet may costto much in delay.

● Skip flow control– Receiver should be a real-time application being

able to cope with the (bounded) rate the sender transmits at.

Also if we have a broadcast scenario no or little upstream communication can be had.


8

Technical Issues

There are two fundamental issues when designing a real-time communications system:

1) Synchronization.Sender and receiver needs to be synchronized to each other to avoid re-occurring buffer over-flows and under-runs

– Essentially having the sender and receiver following a clock at exactly the same frequency.

2) Jitter.Jitter is the varying of the packet-delay around the mean delay. Jitter is usually caused by the network, but could also result from processing stages in the sender that varies in time.

– The larger the jitter the larger the buffer at the receiver-side has to be.


9

Achieving Synchronization 1) Implicit synchronization

The receiver buffer level is used to control the speed of play-out. The most simplistic approach is to throw away suitable input at buffer over-flow and interpolate at buffer under-run

2) Global synchronization

Transmitter and receiver can be part of a global synchronization system like NTP (Network Time Protocol)

3) Point-to-point synchronization

Receiver needs to know from the syntax of the received data stream when certain bits should have been sent in relation to other synchronization point. Using the arrival-times of these bits, the receiver employs a smoothing algorithm, like PLL (Phase Locked Loop) to slave a local clock to the transmitter's system clock.


10

Implicit Synchronization - Example● Assume the receiving buffer is of size N bytes● Assume data is sent isochronously in packets of size

M < N bytes.

N

Buffer utilization

Receiverstartsplay-out

Packets do not arrive at regular time instants = jitter

M

Buffer overflow:Receiver throws awaythis packet!

time Here the discarded packet was supposed to be played-out

Receiver emptiesbuffer too slowly dueto a lower frequencyof the system clock

Packets are movedforward in buffer


11

Point-to-point Synchronization

Example MPEG Systems

27MHz

33bit 90kHzProgram Clock Reference (PCR) 9bits

299000

Coder

MUXCoder

Coder

Coder

188 bytesMPEG Transport Stream (MPEG-TS) at 40 Mbit/s

A PCR time stamp is inserted inheader at least every 100ms. (Once every 1700 packet is enough whenwe have a 40 Mbit/s TS)


12

IETF Audio/Video Transport

The avt working group has defined the Real-Time Transport Protocol (RTP) for multimedia transport (RFC3550)

● Collect common fields for synchronization, packet ordering and data format in one header.

● Different applications may reuse same RTP-stack with respect to the above mentioned parameters

● Thanks to the type identifier, a library or a host may safely determine the format of incoming streams and provide the same service for various formats. – See translation and mixing


13

RTP does not...

...guarantee● Bandwidth● Low jitter

RTP is strictly an end-to-end protocol. Network guarantees on bandwidth, jitter etc is up to separate quality-of-service mechanisms.


14

RTP Terminology

RTP SessionThe set of communicating entities which may via RTP and RTCP know of each other's identities.

Synchronization SourceThe origin of a stream where RTP-packets are generated. Identified by a 32-bit number: SSRC.

TranslationThe act of converting a stream to another formatand/or rate.

MixingThe act of adding two or more streams together and encode the result into a new stream (with new SSRC).


15

RTP Oneway Unicast Example

RTP

RTCP

RTCP

SSRC:1 SSRC:2

SSRC:1 sending a single media stream to SSRC:2

● There will be three streams.● The RTP stream carries the media.● Two RTCP (Real-Time Control Protocol), one in each

direction carries extra control information.● It is recommended that the RTCP traffic only adds an extra

5% to the RTP bandwidth.


16

RTP Translation Example

Terminals only supporting H.263 video

This server is a SIP-proxy and also redirects RTP session ports to go through the server instead (thereby bypassing the firewall). The server can then add

● Privacy by encryption● Translation services. I.e.,

- transcoding of formats not understood by client- bit-rate conversion

firewall

“weak” terminals only supportinglow quality rates.


17

RTP Mixing Example

This machine is mixing together the separate video streams into one video stream which is multicast back to members

SSRC: 2

SSRC: 5

SSRC: 4

SSRC: 12

SSRC: 10CSRC: (2,4,5,12)


18

RTP Header

timestamp

synchronization source identifier (SSRC)

contributing source identifiers (CSRC) (optional)

payload

ver P X CC M payload type sequence number

padding

Fixedpart


19

Synchronization Source Identifier● Every RTP packet is marked with

32-bit number identifying a single timing and sequence number space

● SSRC provides a transport layer independent “tag”. RTP may run over various protocols(Ipv4, Ipv6, ATM)

● SSRC must be unique among all hosts taking part in the RTP session

● SSRC is picked randomly, but sender must in case of a detected collision issue a BYE message over the control channel and change SSRC.

timestamp



payload


padding


20

CC● When a mixer creates a new

stream out of others it may include the SSRC:s of up to 15 of the contributing sources.

● CC-field denotes the number of source identifiers in CSRC field.

● Using CSRC is a safety measure against loops in a RTP streaming session

Contributing Source Identifiers

timestamp



payload

ver P X M payload type sequence number

padding


21

Payload● The payload type (7 bits) indicate

the format of the media put into the payload area. Type 96 to 127 are dynamic, and specified outside the RTP protocol. Compared to the number of codecs out there one realizes that assigning the payload type from the dynamic range might be common. It is then up to the session handling mechanism to inform parties about the type used.

● A flow can only carry one payload type. But the type can change.

● Each payload type has a reference clock frequency associated with it.

timestamp



payload


padding


22

RTP Payload Types (RFC3551)PT A/V Clock Freq Channels MIME-type Description

3 audio 8000Hz 1 audio/gsm 13.2kbps speech coder10 audio 44100Hz 2 audio/l16 two-channel CD-style11 audio 44100Hz 1 audio/l16 one-channel CD-style14 audio 90kHz special audio/mpa MPEG audio layer 1,2,326 video 90kHz - video/jpeg Motion-JPEG32 video 90kHz - video/mpv MPEG-1,2 elementary stream33 a/v 90kHz special video/mp2t MPEG-2 transport stream 34 video 90kHz - video/h263 H.263 video telephony

35-71 unassigned72-76 reserved77-95 unassigned96-127 dynamicUpcoming:

AMR / AMR-WB speech codec used in 3G mobile systemMotion-JPEG2000H.264 video codec (The new MPEG-4 AVC profile)DV audio/videoMPEG-4 Elementary streams etc. etc.


23

Timestamp● 32-bit resolution

● Indicates the sampling instant of first octet of payload data.

● The mapping from transmitter's reference clock to timestamp is defined by payload type.

● Initial value is random

● Clock resolution should be enough to support synchronization and jitter measurements based on timestamp alone.

timestamp



payload


padding

● Note that sender also transmits timestamps of “wall clock time” in RTCP sender reports. These are needed to synchronize two or more streams with each other


24

Sequence Number● Each packet has a 16-bit

sequence number

● Increases with one for each new sent packet.

● Starts with a random value

● Used to detect packet loss. Need not be related to play-out order

● Basic RTP does not provide error correction

– FEC extensions exist

– limited ARQ profile exist

timestamp



payload


padding


25

Marker bit● Marks this packet as special in a sense

defined by payload type.

– May be set in speech formats for indicating talk starts again after a period of silence (or comfort noise)

– Most video formats specify the marker bit to be set to “1” in the last packet for a frame. This gives a simple indication that it is time to display.

timestamp



payload


padding


26

RTP Embedding● RTP is designed to run over any packet switched

network. On Internet we use UDP/IP.

payloadIP-header UDPheader

RTPheader

20 8 12

The overhead per RTP packet will be 40 bytes. This is sometimes seen as too excessive and several header compression techniques have been suggested:

● RObust Header Compression (ROHC) – RFC3095● Enhanced Compressed RTP (CRTP)- RFC3545● IP Header Compression – RFC2507 (LuTH)


27

RTP Interleaving

How to mix audio and video?● Same session, different SSRC

– Does not support different QoS profiles– Impossible in a multicast scenario to join just the

audio stream (for instance)● Different sessions, same SSRC

– works ok. SSRC:s totally unrelated● Different sessions, different SSRC

– works ok. SSRC:s totally unrelated


28

Real-time Transport Control Protocol (RTCP)

Some of the functions of the sister-protocol RTCP are:● Provide feed-back of the reception quality● Carry a persistent identifier for a RTP source (CNAME),

something that is fixed for each related stream.● Carry NTP timestamps enabling inter-media synchronization.● Keep track of number of participants so that RTCP can scale

(i.e., lower the rate at which control messages is sent)● Optional minimal session control, e.g. names of participants.

Note: If RTP data is sent to the UDP-port P (should be even) RTCP messages should be sent on port P+1.


29

RTCP Compound Packets

Several RTCP packets are concatenated to form one compound packet. These will at least contain two individual packets (SR/RR and a SDES).

Sender OR Receiver Report (SR/RR)

(Additional Receiver Reports). . .

.

Source Description (SDES)

BYE or APP packet (optional)

There is a also an encryption option which is skipped here


30

RTCP Sender Report (SR)

SSRC of sender

NTP timestamp (most significant word)

ver P RC PT = SR = 200 length

NTP timestamp (least significant word)

RTP timestamp

sender's packet count

sender's octet count

SSRC_1 (SSRC of first source)

extended highest sequence number received

inter-arrival jitter

last sender report NTP timestamp (middle 32 bits)

delay since last sender report (65536Hz accuracy)

fraction lost cumulative number of packets lost

No of receiver reports

senderinfo

Receiverreport block 1


31

RTCP Receiver Report (RR)

SSRC of sender

ver P RC M PT = RR = 201 length

SSRC_1 (SSRC of first source)

extended highest sequence number received

inter-arrival jitter

last sender report NTP timestamp (middle 32 bits)

delay since last sender report (65536Hz accuracy)

fraction lost cumulative number of packets lost

Receiverreport block 1

When participant in an RTP-session only receives flow we skip the sender information block. Note that a receiver need to have an SSRC anyway! A problem here is to which address and port do we send our RTCP messages! If it's a multicast session on UDP(IP#, port) we send them to UDP(IP#,port+1), but if it's a unicast session the RTCP upstream IP# and port has to be configured by a higher level session protocol.


32

RTCP Source Description (SDES)

ver P SC M PT=SDES=202 length

SSRC / CSRC_1

SDES items. . .

No of SSRC/CSRC chunks

SSRC / CSRC_2

SDES items. . .

Chunk1

Chunk2

SDES information blocks must be sent by every sender. It gives some textual information regarding the flow. The flow might possibly be the result of mixing in which case information for each mixed flow is present. An SDES item looks like

item code length UTF-8 string


33

SDES ItemsCNAME (1) Mandatory Item.

Should uniquely identify the source. Name should be algorithmically derived from e.g., host IP.Flows from different RTP sessions but with the same CNAME indicates inter-media synchronization.

NAME (2) User Name. Ex: “John Doe, Bit Recycler”

EMAIL (3) Electronic mail address according to RFC2822.

PHONE (4) Phone number with a “+” replacing int. access code.

LOC (5) Geographic user location. Ex: “Room 473 A-house”.

TOOL (6) Name and optionally version of application generating the flow

NOTE (7) Transient message describing state of source.

PRIV (8) Experimental or application specific. Has another syntax for the “UTF-8 string” field.


34

RTP Synchronization 1

SSRC: 1

referenceclock RC1

referenceclock RC2

SSRC: 2

referenceclock RC3

SSRC: 3

NTP global cloud“wall clock time”

receiver

RTCP SR packets determines the RTPtimestamps' relation to the global wall clocktime. Synchronization between (1,2) and (3) may be inexact due to bad NTP inaccuracy.

RTCPSSRC: 1(NTP1, RTP1)



Global Synchronization

Different RTPsessions on samehost


35

RTP Synchronization 2

SSRC: 1

referenceclock RC1 reference

clock RC2

SSRC: 2

SSRC: 3

receiver

Receiver sees that CNAME:s arethe same for SSRC:1 and 2.Receiver then knows these flows use the same reference clock and can achieve point-to-point synchronization with RC1 using timestamps from both flows. SSRC: 1 and 2 may be played out in absolute synchronization!

RTCPSSRC: 1(NTP1, RTP1)CNAME: a.b.c.d

Inter-media synchronization in absence of global synchronization.

RTCPSSRC: 2(NTP1, RTP2)CNAME: a.b.c.d

RTCPSSRC: 3(NTP3, RTP3)CNAME: e.f.g.h


36

Software with RTP support

Interested in software for streaming media?

Check out for instance,● JMF 2.0 (Java Media Framework)● Real Helix Player (Open Source) (helixcommunity.org)● UCL Common Multimedia Library

(www-mice.cs.ucl.ac.uk/multimedia/software/common/)● JRTPLIB (research.edm.luc.ac.be/jori/jrtplib/jrtplib.html)

● Live.COM Streaming Media (www.live.com/liveMedia/)● ... many more player/streaming projects out there...


37

Network Time Protocol (NTP)● www.ietf.org/rfc/rfc1305.txt (NTP version 3)● Is not nicely formatted. Check out PDF:s at

www.eecis.udel.edu/~mills/database/rfc/rfc1305/

Purpose:

Keeping hosts on the Internet synchronized to the Coordinated Universal Time (UTC)

– UTC has leap seconds so that the Sun crosses the Greenwich meridian at 12:00:00 +/- 0.9 seconds.

– Leap seconds decided by the International Earth Rotation Service. (33 so far)

– UTC differs with an integral amount of seconds to the International Atomic Time (TAI), a mean of more than 200 atomic clocks around the world...


38

NTP cont.

Builds a hierarchy of hosts synchronized to the imaginary “Stratum 0” clock.

Stratum 1

Stratum 2

Stratum > 2

Via GPS

Atomic Clock


39

NTP cont.● Many modes of operation

– Client to lower stratum hosts (Client/Server)– Peering with various hosts (Symmetric mode)– Broadcast/multicast client on LAN:s

● Typically an NTP clock is within 20ms of its synchronization source (Mills et. al. 1997)– Is this good? (Imagine a pair of IP-driven

speakers!)


40

Session Description Protocol (SDP)● RFC2327● MIME-type: application/sdp● SDP conveys information needed to join one or more

multimedia sessions– Session name and purpose– Time(s) when session is active– What media is involved– Information on how to receive those media

(addresses, ports, formats etc.)– Optionally bandwidth requirements– Optionally how to contact person responsible


41

SDP Syntax (selected parts)

Information is in human readable form coded using UTF-8 encoded Unicode character set.

Session description v= (protocol version) o= (owner/creator and session identifier). <username> <session id> <version> <network type> <address type> <address> s= (session name) i=* (session information) <more optional fields> <One or more time descriptions> a=* (zero or more session attribute lines)

Time description t= (time the session is active) r=* (zero or more repeat times)

Media description m= (media name and transport address) i=* (media title) c=* (connection info - optional if included at session-level) b=* (bandwidth information) k=* (encryption key) a=* (zero or more media attribute lines)


42

SDP Example v=0 o=mhandley 2890844526 2890842807 IN IP4 126.16.64.4 s=SDP Seminar i=A Seminar on the session description protocol u=http://www.cs.ucl.ac.uk/staff/M.Handley/sdp.03.ps [email protected] (Mark Handley) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 11 m=video 51372 RTP/AVP 32 m=application 32416 udp wb a=orient:portrait

Note that an SDP-session is not the same as an RTP session. The three medias above will flow into three parallel RTP sessions on different UDP ports but to the same (multicast) address.


43

Real-time Streaming Protocol (RTSP)

● RFC2326● “Network remote control for multimedia servers”● Mimics HTTP. Example of the rtsp: URI:

rtsp://videoserv.liu.se/filmtest/audio● Well-know port 554.● RTSP is state-ful while HTTP is state-less. An RTSP session

typically contains state such as:

– One of {Init, Ready, Playing, Recording}

– PLAY-commands are queued at server

– A PAUSE may be set to appear at a specific time

for many URI:s at the same time!


44

RTSP Client to Server MethodsSETUP Agree on how a single media should be streamed.

Addresses, protocols etc. (in an on-demandscenario also ports for upstream RTCP messages)

DESCRIBE Request session description data

PAUSE Pause the indicated URI

PLAY Start play-back of indicated URI

RECORD Tell server to start record from previouslygiven invitation.

SET_PARAMETER Generic method for setting stream parameters

GET_PARAMETER Generic method for reading stream parameters

ANNOUNCE Send session information for later recording

OPTIONS Query URI for supported methods

TEARDOWN Stop streaming and free resources


45

RTSP Session SetupA client may connect to a RTSP server and query different URI:s via DESCRIBE-calls.

When client first issues a SETUP-call for a certain media URI an RTSP-session is initiated,

Client->S: SETUP rtsp://foo/twister/video RTSP/1.0 CSeq: 3 Transport: RTP/AVP;unicast;client_port=8002-8003

Server->C: RTSP/1.0 200 OK CSeq: 3 Transport: RTP/AVP;unicast;client_port=8002-8003; server_port=9004-9005 Session: 12345678

This is indicated by the Session: field in the server response. The client should in future requests include this in header. The session identifier should be a randomly created string with at least 8 characters.


46

SDP Pointing out RTSP

An SDP session description might point out media as a set of RTSP controlled resources.

v=0o=- 2890844256 2890842807 IN IP4 204.34.34.32s=I came from a web paget=0 0c=IN IP4 0.0.0.0m=video 8002 RTP/AVP 31a=control:rtsp://rtspserv.com/movie.audm=audio 8004 RTP/AVP 3a=control:rtsp://rtspserv.com/movie.vid

Note that the m= sections do not convey any address/port information for how media is streamed. This is left to the SETUP-call. This kind of SDP-data could be given from a HTTP-request but can also be the result of a DESCRIBE-call for a specific RTSP URI.


47

RTSP PresentationEach rtsp: URI correspond to a presentation. A presentation is one of two things

● A single media

● A container file consisting of one or more submedias

To find out which a client can

● Issue a SETUP call and prepare to get back an error “403 Aggregate Operation Not Allowed”.

● Issue a DESCRIBE call and study the SDP m= fields. Although note that DESCRIBE-support in an RTSP-server is not a “MUST”!

For playing an aggregated media a separate SETUP call per submedia is needed.

The RTSP-server could support aggregate control (PLAY, PAUSE etc.) of all submedias. One then issues the method on the container URI.


48

RTSP Example (Multicast Live Audio)C->S: DESCRIBE rtsp://live.example.com/concert/audio RTSP/1.0 Cseq: 1

S->C: RTSP/1.0 200 OK Cseq: 1 Content-Type: application/sdp Content-Length: 44

v=0 o=- 2890844526 2890842807 IN IP4 192.16.24.202 s=RTSP Session m=audio 3456 RTP/AVP 0 a=control:rtsp://live.example.com/concert/audio c=IN IP4 224.2.0.1/16

C->S: SETUP rtsp://live.example.com/concert/audio Cseq: 2 Transport: RTP/AVP;multicast


49

RTSP Example cont.

S->C: RTSP/1.0 200 OK Cseq: 2 Transport: RTP/AVP;multicast;destination=224.2.0.1; port=3456-3457;ttl=16 Session: 0456804596

C->S: PLAY rtsp://live.example.com/concert/audio Cseq: 3 Session: 0456804596

S->C: RTSP/1.0 200 OK Cseq: 3 Session: 0456804596


50

Streaming Through a Firewall

In general UDP-based traffic is tougher to handle for firewalls since there is no transport-level connection mechanism like TCP's which the firewall could intercept. Several solutions exists however:

● A firewall might work on application-level and intercept SETUP-calls and open up for incoming flows.

● RTSP includes an option for embedding RTP/RTCP packets over TCP

● One could deploy an RTSP proxy beside the firewall and instruct the client to always go via the proxy. Such a proxy instructs the real RTSP-server to stream to the proxy server instead.


51

IETF Working Groups

avt – Audio/Video Transport– RTP / RTCP– Payload formats for various audio and video

formats (µ-law/linear PCM, H.263, MPEG, GSM..)

mmusic – Multiparty Multimedia Session Control– SDP– RTSP– SIP (earlier versions before SIP got a working

group of its own)


52

Summary● Real-time requires synchronization.● Jitter affects buffer size.● RTP and RTCP offers a common framework for

handling– Synchronization– Packets arriving out of order– Source Identification– Loop detection

● SDP is a textual syntax for describing session setup parameters.

● RTSP is the “remote VCR” control protocol.

Documents

TSIN02 - Internetworking · TSIN02 - Internetworking 8 Technical Issues There are two fundamental issues when designing a real-time communications system: 1) Synchronization. Sender