Advances in Network-adaptive Video Streaming

Advances in Network-adaptive Video Streaming

Bernd GirodBernd GirodJ. Chakareski, M. Kalman, Y. J. Liang, E. Setton, R. ZhangJ. Chakareski, M. Kalman, Y. J. Liang, E. Setton, R. Zhang

Information Systems LaboratoryInformation Systems LaboratoryDepartment of Electrical EngineeringDepartment of Electrical Engineering

Stanford UniversityStanford University

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 2

Streaming media: a huge success

Hundreds of thousands of streaming media servers deployed

> 1 million hours of streaming media content produced per month

Hundreds or millions streaming media players RealPlayer

– Most popular Internet application second only to Internet Explorer [Media Metrix]

– More than 400 million unique registered users– More than 200,000 new users per day– Open source code


Best-effort packet network• low bit-rate • variable throughput• variable loss• variable delay

Challenges• compression • rate scalability• error resiliency• low latency

Challenges• compression • rate scalability• error resiliency• low latency

Internet Media Streaming

Streaming client

DSL

56K modem

Media Server

Internet

wireless


Outline

What is network-adaptive video streaming?– Better delivery of video packets by considering source

coding, signal processing, and packet transport jointly– Application-layer joint source-channel coding techniques

for the Internet

This talk: review recent advances in 1. Adaptive media playout

2. Rate-distortion optimal packet scheduling

3. Network-adaptive packet dependency management


Adaptive Media Playout

Fixed deadline Flexible deadline

Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions

Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions

5% packet loss2 sec average receiver buffer

5% packet loss2 sec average receiver buffer

[Steinbach, Färber, Girod, ICIP 2001]


Modification of Playout Speed

Video: adaptation of display rate Audio and speech: Stretching based on time-domain interpolation

algorithm WSOLA [Verhelst et al., 1993, Liang 2001]

Output packet

1/20/1 2/3 3 4

Original packet

Pitch-period

0 21 3 4

Template


Audio Speed Adjustment

Waveform Similarity Overlap Add (WSOLA) method allows speed adjustment without changing pitch

Speech demo

Music demo

originalslower 30% faster 30%

originalslower 30% faster 30%


Reduced Pre-roll Time for Stored Streams

G=1.092, B=0.42, TG=20 sec, TB=2 sec, TRTT=220 msG=1.092, B=0.42, TG=20 sec, TB=2 sec, TRTT=220 ms

Probabilityof buffer

underflow< 1%

Probabilityof buffer

underflow< 1%

[Kalman, Steinbach, Girod, ISCAS 2002]


Rate Scalability by Playout Speed Adjustment

Server

20 kbps

50 kbps

95 kbps

25 kbps

55 kbps

100 kbps

Channel(mean throughput)

85 kbps

(29.2 dB, 53.3 kbps)

(33.1 dB, 93.6*0.9=84.2 kbps)


First Things First: Smart Prefetching

Idea: Send more important packets earlier to allow for more retransmissions

Server ClientInternetInternet

Request stream

Request stream

Rate-distortionpreamble

Rate-distortionpreamblePrefetch

times

Prefetchtimes

Video dataVideo data

Repeatrequest

RepeatrequestRepeatrequest

RepeatrequestRepeatrequest

Repeatrequest


Streaming as a Packet Scheduling Problem

Which media units should be selected for transmission, and when?

Requirements– Meet rate constraint– Meet latency constraint– Maximize reconstruction quality

Rate-distortion framework proposed, e.g., in [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001]

time

pre-encoded media unitstransmissionopportunities

server


Markov Decision Tree for One Packet

... N transmission opportunities before deadline

send: 1

ack: 1

0

0

0

send: 1

0

send: 1

0

ack: 1

01

01

0

0

1

1

1

0

0

0

0

tcurrent tcurrent+t tcurrent+2t

Action Observation

“Policy“ minimizing

J = D + R“Policy“ minimizing

J = D + R


Packet Delay Jitter and Loss

delay

pdf

lead-time

lossprobability

lead-time

lossprobability

loss


Source Description

Each media packet n is labeled by− Bn — size [in bits] of data unit n

− dn —distortion reduction if n is decoded

− tn — decoding deadline for n

P PI

I

B B B P P PI

I

B B B P

A

…

…

…

A …


PB

Source Description

Each media packet n is labeled by− Bn — size [in bits] of data unit n

− dn —distortion reduction if n is decoded

− tn — decoding deadline for n

P PI

I

B B P PI

I

B B B P

A

…

…

…

A …

For video: dn must be made“state-dependent” to accurately capture concealment

For video: dn must be made“state-dependent” to accurately capture concealment


R-D Optimized Streaming Performance

40 60 80 100 120 14024

25

26

27

28

29

30

31R-D OptimizedPrioritized ARQ

Foreman 120 frames 10 fps, I-P-P-… H.263+ 2 Layer SNR

scalable 20 frame GOP Copy Concealment 20 % loss forward

and back Γ-distributed delay

– κ = 10 ms– μ = 50 ms– σ = 23 ms

Pre-roll 400ms

Foreman 120 frames 10 fps, I-P-P-… H.263+ 2 Layer SNR

scalable 20 frame GOP Copy Concealment 20 % loss forward

and back Γ-distributed delay

– κ = 10 ms– μ = 50 ms– σ = 23 ms

Pre-roll 400ms

PS

NR

[dB

]

Bit-Rate [kbps]


R-D Optimized Streaming with a Proxy Server

MediaServer

Switch/Router

ProxyServer

Client

Backbone Network

Packet

Last hopReceiver-driven RaDiO streaming

Sender-drivenRaDiO streaming

Buffer packets

[Chakareski, Chou, Girod, Asilomar 2002, MMSP 2002]


RaDiO Edge: Experiment

40 60 80 100 120 140 160 18027

28

29

30

31

32

33

34

Ave

rage

Y-P

SN

R (

dB)

Bit rate (Kbps)

RaDiO EdgeSender-driven

Video: Foreman, QCIF, 130 frames Compression: H.264 3-layer temporal scalability 72…144

kbps [Liang, 2002] Backbone

– Packet loss rate: 10%

– Delay: shifted -distribution Last hop

– Packet loss rate: 1%

– Delay: shifted -distribution


Streaming with Diversity

MediaServer

Client

Channel 1

Channel 2

Channel N

MediaServer 1

Client

Channel 1

Channel 2

Channel N

MediaServer 2

MediaServer N

Packet Path Diversity Server Diversity


R-D Optimized Streaming over 2 Channels

30 40 50 60 70 80 90 10024

25

26

27

28

29

30

Bit rate (Kbps)

Ave

rage

Y-P

SN

R (

dB)

Over 2 channelsOver 1 channel

Video: Foreman, QCIF, 130 frames

Compression: H.263+ 2-layer SNR scalability 32/64

kbps 2 identical, independent 2-state

Markov channels Good/bad packet loss rates:

3%/15% -distributed delays short/long

Video: Foreman, QCIF, 130 frames

Compression: H.263+ 2-layer SNR scalability 32/64

kbps 2 identical, independent 2-state

Markov channels Good/bad packet loss rates:

3%/15% -distributed delays short/long

[Chakareski, Girod, DCC 2003]


R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD)

50 100 150 200 250 300

23

24

25

26

27

28

29

30

31multiple deadlinessingle deadline

Latency: 100 ms RTT: 100 ms

3 dB

47 %

PS

NR

, in

dB

Bit-Rate, in kbps




Latency: 100ms

Multiple DeadlinesRate: 68.8 kbps

Mean PSNR: 27.0 dB

Single DeadlineRate: 89.0 kbps

Mean PSNR: 23.9 dB


Network-adaptive Packet Dependency Management

Very low latency: no retransmissions Highly robust compressed representation by network-adaptive

management of packet dependencies Utilize ACK/NACK in source coder H.263 RPS, MPEG-4 NEWPRED, H.264 multiframe prediction

Time

Transmission error

NACK


Error Resilience vs. Coding Efficiency

P1

P2

I

P5

230 frames of Foreman coded using H.26L TML8.5. Average PSNR=33.4dB


Rate-Distortion Optimal Reference Picture Selection

( )

1

1,2,... ,( ) arg min ( )

L n

v vl vll

vv v

opt v V v

D p D

J D R

v n J n

[Wiegand, Färber, Girod, 2000]

[Liang, Girod, 2002]


RD Performance of Optimal Reference Picture Selection

LTM buffer: V=5 frames

Feedback round-trip time: dfb=7 frames

Packet loss rate: p=10% Comparison with P-I scheme, where each NACK triggers insertion of I-frame

[Liang, Girod, 2002]


ORPS Performance over Time Axis


MaD Sequence at 10% packet lossNo Retransmissions

Optimal Reference Picture SelectionRate: 320 kbps

Mean PSNR: 39.2 dB

Adaptive I-Frame InsertionRate: 320 kbps

Mean PSNR: 38.2 dB


Conclusions

Network-adaptive video streaming: jointly optimize compression, error control, packet transport, and decoding

Adaptive media playout: “real-time” more flexible than we thought

RaDiO streaming can provide virtual priority mechanisms Proxy servers with RaDiO can improve streaming performance Path Diversity can improve streaming performance RaDiO streaming with multiple deadlines for accelerated

retroactive decodingof late retransmissions (“catch-up decoding”)

Packet dependency management allows robust representations for streaming w/o retransmissions

Documents

Advances in Network-adaptive Video Streaming