QoE Metric and Cross-Layer Optimization for Video over Wireless Networks

1 Z. Li, 2012

Zhu Li

Outline

• Introduction• Research Motivations• A brief overview of my research

– Video QoE metric– Multi-Access Wireless Video Streaming– Wireless Video Broadcasting– P2P Video Networking

• In depth discussion

2 Z. Li, 2012

• In depth discussion – Multi-Access Video Networking,

» Temporal QoE metric and VLBR video adaptation, » NUM formulation and resource pricing scheme» Application in Multi-Access Video Networking» Simulation Results

– Source Channel Coding in Video Broadcasting (extra)

• Summary & Questions

About Me: http://users.eecs.northwestern.edu/~zli

• Bio:– Sr. Staff Researcher, FutureWei Technology, Bridgewater, NJ. – Asst Prof, HK Polytechnic Univ, and CTO, Mudi Technology,

2008.04~2010.10.– Senior, Senior Staff, and then Principal Staff Researcher, Multimedia

Research Lab, Motorola Labs, USA, 2000-08.– Software Engineer, CDMA Network Software Group, Motorola CIG, USA,

1998-2000. – PhD in Electrical & Computer Engineering, Northwestern University,

3 Z. Li, 2012

– PhD in Electrical & Computer Engineering, Northwestern University, USA, 2004.

– IEEE Senior Member, elected vice chair, IEEE Multimedia Communication Tech Committee , 2008~2010.

• Research Interests:– Video Coding and Adaptation, Optimization and Distributed Computing in

Video Networking with applications in mobile TV, wireless video on-demand streaming, and P2P video networking.

– Image/Video Analysis, Machine Learning and applications in Scalable large video repository search and mining problems.

2012

4 Z. Li, 2012

Devices

• Explosive growth of devices:– Billions of cell phones/PDAs– Billions of computers– Billions of TVs– Billions of Media Players

• Different Multimedia

5 Z. Li, 2012

• Different Multimedia Capabilities in:– display, – capture, – storage,– computing, – communication

Networks

• Better technology from equipment makers– Better wireless spectrum efficiency,

WiMAX/LTE– High speed DLS/Cable, 100x100– Fiber optical solutions, GPON

• More capacity from service providers

6 Z. Li, 2012

providers– More bandwidth, better coverage, – Convergence of data, voice and

media service from service providers

– Vertical integration of application and services

Content

• Explosive growth of digital media– Web, Email, Audio, Video, Game– News, Music, Movie, Talk show,

Game, 2nd Life.

• Rapid changes in the way contents are produced and consumed

7 Z. Li, 2012

consumed– Personal vs Commercial– Passive (TV) vs Interactive (Blog,

Game)– Centralized vs P2P

Key Challenges

• Application Needs:– Good Access, be able to get what you want, a storage and communication

problem– Mobility across devices and access sessions: anywhere, on any device,

not tied to a single device/location, get what they want, with good media quality (coding) and availability (communication/networking).

– Intelligence and Personalization: be able to utilize the “big data” collected from both networks and applications, and drive more intelligent and

8 Z. Li, 2012

from both networks and applications, and drive more intelligent and personalized applications

Technology Gap

Technology Gap ?– Video Communication:

» Rapidly widening gap between wireless capacity and explosive growth of mobile video traffic, 14 times in 2015 according to Verizon.

» Re-engineering the Internet for video traffic– Video Storage:

» Storage of multimedia content in cloud – explore various error-resilience and rate-distortion tradeoff characteristics to enable differential storage service.

– Video Computing:

9 Z. Li, 2012

– Video Computing:» Web scale multimedia analysis, indexing and retrieval, » Mobile search

Research and Biz Opportunities

Communication is Computing:– Computing and storage are cheaper than wireless spectrum. Use

computing and storage to mitigate the wireless gap.– Smarter networks: cognitive radio, cross layer design, employing richer

video QoE metric that enhances traffic elasticity and robustness– Smart caching and on-demand transcoding and adaptive streaming within

the network– Emerging standardization effort, eg. MPEG MMT.

10 Z. Li, 2012

Relevant Research Projects Highlights

11 Z. Li, 2012

Project Highlights – QoE Metrics and VLBR Video

• Video Summarization for VLBR video streaming– Developed a frame drop induced distortion metric– Offering a content aware way of temporal distortion adaptation

scheme that can drive the operating bit rate for QCIF video down to PCM voice rate range

– Dynamic Optimization in frame selection

35

36

VLBR Video Examples: R−Davg−Dmax−PSNR Performance

40

50

60summary frames

12 Z. Li, 2012

10 20 30 40 50 60 7029

30

31

32

33

34

35

R (kpbs)

PS

NR

(dB

)

Davg=31.9, Dmax=79.9D

avg=26.1, D

max=59.9

Davg

=49.4, Dmax

=99.1

10 20 30 40 50 60 70 80 90 100 110 1200

20

40

60

80

d(f k,

f k)

summary distortion

λ= 16.0e-4, D(S)=24.6, R(S)=80.8kb

0 20 40 60 80 100 1200

10

20

30

40

d(f k,

f k-1)

Project Highlights – Video Communication

• Intelligent Mobile TV– Supporting highly elastic and robust

QoS for a variety of mobile terminals with graceful visual quality degradation with channel conditions

– Practical frame-size and frame rate adaptation for wide codec support.

– PHY layer adaptation through DE-STC (diversity embedding space-time coding) to induce a set of embedded

13 Z. Li, 2012

coding) to induce a set of embedded channels that best suit the multicast group channel distribution

– APP layer source-channel coding optimization with layered video and digital fountain code

– Targeting in-band, or dual-band (WiFi + 3G) wireless infrastructure.

Project Highlights – Video Communication

• Next Generation Content Networks – Video accounts for > 70% of internet traffic now– P2P video networking, utility based on minimizing freezing in playback,

developed a gradient based solution. (collaboration with Profs. Chiang and Calderbank at Princeton).

– Optimization and distributed computing solutions in IPTV video networking, Primal-Dual decomposition, resource pricing schemes for multi-access and IPTV video networks, (RGC New Staff Grant, in collaboration with Prof. Mung Chiang’s EDGE lab at Princeton)

14 Z. Li, 2012

Prof. Mung Chiang’s EDGE lab at Princeton)– Video TCP: TCP re-engineering for content delivery networks. Reconsider

the congestion measure and pricing as well as source adaptation schemes in TCP to better suit for content intensive traffics.

– Caching and Network Coding schemes for video sharing in Mesh Networks (in collaboration with Prof Cao at HK PolyU).

Project Highlight - Mobile Search• Video Based Mobile Location Search:

– Location search by mobile video capture and query

– Video SIFT points indexing with appropriate scale and spatio-temporal quality metrics

– Fast search with multi-indexing of SIFT point sets. – See my recent ACM MM paper for more detail.

• Image/Video Based Mobile Product Search :– Query-by-Capture, when shopping in the malls,

15 Z. Li, 2012

– Query-by-Capture, when shopping in the malls, just took a picture/video of the product and search the online stores to compare prices.

– Novel Image Fingerprint indexing solution– Gone commercial with dangdang.com (amazon in

China) , can show a demo if interested.

Temporal QoE Metrics and Resource Pricing Control i n Multi-Access Wireless Video Networking

16 Z. Li, 2012

Mixed Voice/Video over CDMA Up Link

Radio tower

• Mixed QoS requirements for Video/Voice traffics

• Limited resource, video has to operate at VLBR

• Shared radio resource, and interference limited capacity, Ri=f(Pi;P-i),

17 Z. Li, 2012

Ri=f(Pi;P-i), • Diversity of channel gains and

source rate-distortion characteristics among users.

• How to optimize video adaptation and transmission to achieve better QoS and radio resource efficiency ?

A General Formulation

• Total utility maximization subject to a shared resou rce constraint,

– Where utility function Ui() is a concave differentiable function reflecting the

max,,,

..,)(max21

xxtsxUi

ii

iixxx n

≤∑∑L

18 Z. Li, 2012

– Where utility function Ui() is a concave differentiable function reflecting the quality-bit rate/resource trade-offs. (true for most video source’s PSNR-R function)

– Difficult to solve the primal problem by allocating {xi} directly, because of coupling of {xi} in constraint.

– Difficult to have all utility information for all mobile users, – Transform the problem for a distributed solution, utilizing computing

capability at mobiles

Distributed Solution of the Dual Problem

•Lagrangian relaxation:

•The dual problem:

•Decomposed into n separable video adaptation problems at mobiles :

]}))(([max{min max,,,0 21

xxxUi

iiixxx n

λλλ

+−∑≥ L

))((max xxxU λλ +−∑

)()(),,,,( max21 xxxUxxxLi

ii

iin −−= ∑∑ λλL

19 Z. Li, 2012

•And a base station resource pricing problem:

))((maxarg*

))((max

))((max

,,,

max,,,

21

21

iiix

i

iiii

xxx

iiii

xxx

xxUx

xxU

xxxU

i

n

n

λ

λ

λλ

−=⇔

−⇔

+−

∑

∑

L

L

∑≥i

ig )(min0

λλ

)()()( max** xxxUg iiii −−= λλ

BTS Mobile i

Announce resource price in iteration kkλ

Mobile optimization:

Protocol for Distributed Optimization

))((maxarg* ik

iix

i xxUxi

λ−=

Report back resource used xi* in iteration k

Increase price, if Otherwise decrease price

∑ >i

i xx max

kkk αλλ ±=+1 Mobile optimization:

))((maxarg* 1k xxUx +−= λ

20 Z. Li, 2012

))((maxarg* 1i

kii

xi xxUx

i

+−= λ

Distributed Optimization for Multiple Access Video Network

• Geometrical Interpretation on price:– From the Karush-Kuhn-

Tucker (KKT) condition:

– Allocations {x *} will have

*** ,,0 λ−=

∂∂

⇒=∂∂

i

i

i x

U

x

L

U1(x1)

U2(x2)

U3(x3)

U1(x1)

U2(x2)

U3(x3)

21 Z. Li, 2012

– Allocations {xi*} will have

the same marginal utility (slope) as -price.

– Optimal price must also be tight on all available resource.

U1(x1)

x1* x2

* x3*

U1(x1)

x1* x2

* x3*

Video Over Multiple Access Channel

• In solving real world problems with this distribute d pricing scheme:– Source coding: scalability, adaptability issues– Diversity in Channel state– Diversity in content– Collaboration in resource allocation, scheduling– Uplink problem: interference limited

22 Z. Li, 2012

– Downlink problem: power limited. – Computational complexity

CDMA Uplink with Mixed Voice/Video Traffic

• Consider a single cell CDMA uplink:– Pvoice – received power for a voice user– M – total voice users– Pvideo – total received power for all video users– Gvoice - modulation scheme related constant, BPSK = 1, QPSK = 2– W - bandwidth (Hz)– voice - voice QoS minimum SINR γ

23 Z. Li, 2012

voice

• Received Power Constraints:– QoS for voice users:

– Max allowable total received power for video users

,)1(0

voicevoicevideo

voice

voice

voice

PMPWn

P

R

WG γ≥−++

.1 0max WnPMR

WGP voice

voicevoice

voice −

−+=

γ

Problem Formulation

• Control video mobiles’ transmitting power to achiev e social optimality in total received utility (video quality ) :

– Optimization is over a sliding window of size T– Utility (PSNR, e.g) is a function of total rate in T

( ){ } ( )( ) ( ) ],0[,..,max max11 0

01

TtPtPtsdttRUN

jj

N

j

T

jjtP

Njj

∈∀≤

∑∑ ∫

==≥≤≤

P

24 Z. Li, 2012

– Utility (PSNR, e.g) is a function of total rate in T– Total N video users.

• How to solve ?– Spend resource that can give maximum return in quality

» Account for content diversity, each has different R-D curves» Account for channel state diversity,

– Distributed solution

Multiple Access for Dominanting Received Power Users

• Time-Division Multiplexing (TDM) is needed among vi deo users– Video users’ received power too strong for spectrum efficiency – Example: 4 video users’ achieve able total rates plot:

2000

2500

3000

Vid

eo u

ser’s

tota

l ach

ieva

ble

rate

(kb

ps)

GREEDYSIMCONST

25 Z. Li, 2012

– Therefore, we choose TDM among video users.

5 10 15 20 25 30 350

500

1000

1500

2000

Number of voice users

Vid

eo u

ser’s

tota

l ach

ieva

ble

rate

(kb

ps)

Problem Formulation with TDM Among Video Users

• Allocate transmission slots among video users to ac hieve social optimality in total received utility (video quality) :

{ } ( ) ,..,~

max11

01

∑∑==≥

≤≤≤

N

jj

N

jjj

tTttstU

Njj

( ) ( ).~jTDMjjj tRUtU =

26 Z. Li, 2012

– Total time slots {tj} length is T. – RTDM is the rate achieved using Pmax for a single video user, with current

voice traffic load.

Dual Decomposition : Pricing Solution

• The primal problem is difficult to solve.– The problem is convex, since we assume utility functions are convex.– Constraints are also convex– Strong duality exists.

• Dual Decomposition through Lagrangian Relaxation:– Lagrangian:

( ) ( ) ,~

,max 0

−−= ∑∑≥

N

j

N

jj TttUJ λλtt

27 Z. Li, 2012

– Mobile source adaptation surplus problem:

– Base station resource pricing problem:

( ) ( ) ,,max11

0

−−= ∑∑==

≥j

jj

jj TttUJ λλtt

( ) ( )( ).~maxarg jjjlj ttUt

jλλ −=

( )( ),,max 0 λλλ tJ≥

( ) ( ).~jTDMjjj tRUtU =

Video Source Adaptation with Resource Pricing

•The source surplus problem is to maximize pay off a s utility minus cost in resource

–Distributed to each video source, interact with other video users thru the price. –If scalable coded source, optimal bit extraction subject to a price on resource. Utility could be the PSNR quality of the video–For VLBR (e.g. 24~120kpbs), code video frames at very low PSNR is not

( ) ( )( ).~maxarg jjjlj ttUt

jλλ −=

28 Z. Li, 2012

–For VLBR (e.g. 24~120kpbs), code video frames at very low PSNR is not preferable. Use video summarization scheme instead.

( ) ( ) ( )jjjS

j StSDSj

λλ += minarg*

Temporal Video QoE Metric and Video Summarization

•What is video summary ?–A shorter version of the original video that preserves most information.

•Definitions:– n-frame video sequence:

– m-frame video summary:

– reconstruction by repeating last summary frame:

},,{110 −

=mlll fffS L

}',','{' 110 −= nS fffV L

},,{ 110 −= nfffV L

29 Z. Li, 2012

– reconstruction by repeating last summary frame:

– frame loss distortion:

– rate:

}',','{' 110 −= nS fffV L

)',()(1

0jj

n

j

ffdSD ∑−

==

+==

∑

∑∑ −

=

−

=−

=−

coding-inter,

coding-intra,

)()(1

1

1

01

0

1

0

m

t

ll

l

m

t

l

m

tl

t

t

t

t

rr

r

fbSR

Video Summary QoE metrics Examples

n=10, S={f0, f3, f5, f8 } , m=4, D(S)=0.6

1 2 3 4 5 6 7 8 9 100

0.2

0.4

0.6

0.8

1

1.2

1.4

d(f k

, fk-

1)

summary frames

f f f f f f f fffV=

30 Z. Li, 2012

1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

d(f k

, fk')

summary distortion

1 2 3 4 5 6 7 8 9 10

f0 f0 f0 f3 f3 f5 f5 f8f8f5

f0 f1 f2 f3 f4 f5 f6 f9f8f7d(f0, f1)

d(f0, f2)

V=

VS’=

Frame Distortion for Summarization: What is a good d(fj, fk) ?

scale PCA.

352x240 video frame 11x8 image icon d-dimensional point

.d(fj,fk)

10

12

31 Z. Li, 2012

“foreman” seq in 2-d (1st and 2nd component) PCA space

1

101 201

301

X1

X2400

50 100 150 200 250 300 350 4000

2

4

6

8

10

frame number k

d(f

k, f’

k)

Surplus problems at mobiles•The adaptation problem:

– compute summary at mobile j, s.t. the following surplus function is maximized,

( ) ( ) ( )

( ) ( )TDM

jj

S

jjjS

j

R

SRSD

StSDS

j

j

λ

λλ

+=

+=

minarg

minarg*

32 Z. Li, 2012

– for the given voice traffic load, RTDM is known, R(Sj) is the bit rate for the resulting video summary.– exhaustive search is exponential in complexity, – the problem has some structure for which we will exploit for a Dynamic Programming solution.

Distortion State and Cost

• Summary Segment Distortion:

• Distortion State Dtk, for summaries with t frames ending with fk,

• Bit cost for Dtk,

∑−

=

++ =

11

1 ),(t

t

t

t

t

l

ljjl

ll ffdG

}{min2

2

1

1

2210

,,

nk

kl

ll

l

lll

kt GGGGD

tt

++++=−

−

LL

−1t

33 Z. Li, 2012

t

• The surplus problem:

∑−

=

=1

0

)(t

jl

kt j

fbR

}{min221 ,,

,

TDM

ktk

tlll

kt

R

RDJ

t

λλ +=−L

The surplus recursion at mobile– To simplify notation, let a new price on bit be,

–The recursion:

++−+++=

++

+++++=

+=

−

−

−−

−++

+

{min

)]}()(

)()([{min

}{min

11

1

1

1

121

121

100,,

11,,

,1

nknnll

kl

nk

kl

l

lll

kt

kt

lll

kt

t

t

tt

t

GGGGGG

fbfb

fbfbGGG

RDJ

λ

λλ

L

L

LL

L

TDMR

λλ =

34 Z. Li, 2012

+−

+−=

++−−

++++=

++++

++−+++=

−

−−

−

−−

−

−

−−

−−−

−

−−−

−

−−

codinginter if},{min

codingintra if},{min

)}(])([

)]()([{min

)}()]()()([

{min

1

11

1

11

1

,1

11

11

1

121

1

111

1

2

1

121

,,

,,

00,,

10

0,,

kl

kllt

l

kkllt

l

k

e

nk

kl

nl

lnl

l

lll

kl

nk

kl

nl

nl

ll

l

lll

t

tt

t

tt

t

ktl

tt

ttt

t

ttt

t

tt

reJ

reJ

fbGGG

fbfbGG

fbfbfbfb

GGGGGG

λ

λ

λ

λ

λλ

λ

λ

44 344 21

LL

L

L

L

L

A Viterbi Like Algorithm

3

3.5

4

4.5

5

summarization: λ = 1.50e-004, Kmax=5

J23=9.89

J24=10.11

J25=9.99

J33=10.00

J34=10.04

J35=9.89

J43=10.09

J44=10.15

J45=10.00

J54=10.24

J55=10.09 J6

5=10.24

fra

me

k

– DP solution for surplus maximization under a given price on resource

– Start with first frame– Compute the max surplus

incoming edge at each

35 Z. Li, 2012

1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60

0.5

1

1.5

2

2.5

J10=9.99

J21=10.09

J22=9.99 J3

2=10.05

fra

me

k

epoch t

incoming edge at each node

– Backtracking for optimal solution.

Summarization Results

60

80summary distortion

λ= 16.0e-4, D(S)=24.6, R(S)=80.8kb

0 20 40 60 80 100 1200

10

20

30

40

50

60

d(f k,

f k-1)

summary frames

60

80summary distortion

λ=12.0e-4 D(S)=16.8, R(S)=107.2kb

0 20 40 60 80 100 1200

10

20

30

40

50

60

d(f k,

f k-1)

summary frames

36 Z. Li, 2012

10 20 30 40 50 60 70 80 90 100 110 1200

20

40

d(f k,

f k)

10 20 30 40 50 60 70 80 90 100 110 1200

20

40

d(f k,

f k)

=1.6e-5, PSNR=30dB, D(S)=24.6, R(S)=80.8kb

=1.2e-5, PSNR=30dB, D(S)=16.8, R(S)=107.2kb

λ λ

Video Summarization + Transcoding Scheme for VLBR Channels

• Optimally select a subset of frames to code at a hi gher PSNR quality

34

35

36

PS

NR

(dB

)VLBR Video Examples: R−Davg−Dmax−PSNR Performance

– “Foreman” sequence

– Bit rate range: 11.2kpbs ~46.5kbps

– PSNR: 29dB ~

37 Z. Li, 2012

10 20 30 40 50 60 7029

30

31

32

33

R (kpbs)

PS

NR

(dB

)

Davg=31.9, Dmax=79.9D

avg=26.1, D

max=59.9

Davg

=49.4, Dmax

=99.1

– PSNR: 29dB ~ 34.3dB

– R(S)

Demo !

Base Station Price Control Problem

• Base station solves for a price that maximizes total utility

– Achieved through a sub-gradient method, checking for constraint violation at each price iteration:

( )( ),,max0

λλλ

tJ≥

( )( ) .,0max1

1

−+= ∑

=

+ TStN

j

ijj

iii λαλλ

38 Z. Li, 2012

– The sub-gradient search converges if the step sizes:

– In practice, price iteration stops when total utility improvement ratio is below certain threshold.

– Also the time slot allocation need to be schedulable.

1 ∑

=j

0lim =→∞i

i α ∞→∑i

iα

Joint Packet Scheduling for video summary transmission

• Video packets are delay sensitive. – In TDM scheme, we have a GREEDY solution: sort packets by their

deadlines, transmit the nearest deadline ones.

0 10 20 30 40 50 60 70 80 900

5

P1(t)

Received Powers under GREEDY

0 10 20 30 40 50 60 70 80 900

5

P2(t)

39 Z. Li, 2012

– Pricing iteration is actually on schedulability (deadline violations)

0 10 20 30 40 50 60 70 80 900

0 10 20 30 40 50 60 70 80 900

5

P3(t)

0 10 20 30 40 50 60 70 80 900

5

P4(t)

Frame k

( ){ }}0,max{,0max1 iGREEDYii λβλλ ∆+=+

Simulation Results

• Simulation set up:– Channel (IS-95 alike):

Entity Symbol Value Bandwidth W 1.228MHz

Noise density n0 8.3*10-7 mW/Hz

Voice target SINR γvoice 6dB Voice modulation BPSK

Voice received power Pvoice 1mW Voice spreading gain Gvoice 128

Voice rate R 9.6kbps

40 Z. Li, 2012

Video Users:» 4 segments (90 frames each) from “foreman” and “mother-daughter”

sequences» Fixed PSNR: 27.8dB (foreman), and 31.0dB (mother-daughter)

Voice rate Rvoice 9.6kbps Video target SINR γvideo 6dB Video modulation QPSK

Price and Distortion Convergence

• Pricing iteration convergence at base station (left Fig), and summarization distortions at mobiles (right Fig):

0.05

0.06

0.07

0.08

0.09

0.1

Pric

e

15

20

25

Dis

tort

ion

Per

Fra

me

User 1User 2User 3User 4

41 Z. Li, 2012

1 2 3 4 5 60

0.01

0.02

0.03

0.04

Iterations

Pric

e

1 2 3 4 5 60

5

10

Iterations

Dis

tort

ion

Per

Fra

me

Simulation Results

• Resulting video summaries with pricing co-ordinatio n– D(S1)=3.09, D(S2)=6.42– D(S3)=0.76, D(S4)=0.81

10 20 30 40 50 60 70 80 9005

1015

D(S

1)

Summary Distortion

1015

2)

42 Z. Li, 2012

10 20 30 40 50 60 70 80 9005

10

D(S

2)

10 20 30 40 50 60 70 80 9005

1015

D(S

3)

10 20 30 40 50 60 70 80 9005

1015

D(S

4)

Frame k

Simulation Results – Compare with SIMCAST

• Resulting video summaries without pricing co-ordina tion– D(S1)=2.85, D(S2)=31.43– D(S3)=0.059, D(S4)=0.068

10 20 30 40 50 60 70 80 9005

1015

D(S

1) Summary Distortion under SIMCONST

1015

2)

43 Z. Li, 2012

10 20 30 40 50 60 70 80 9005

10

D(S

2)

10 20 30 40 50 60 70 80 9005

1015

D(S

3)

10 20 30 40 50 60 70 80 9005

1015

D(S

4)

Frame k

Performance Summary

• The Performance for CDMA uplink video networking:– In this work we proposed an efficient solution to support mixed voice and

VLBR video traffic that can help seamless migration from 2.5G to 3G and B3G systems

– The solution is distributed, with minimum communication overhead (prices, summary frames) between base station and mobiles

– The computational complexity for source adaptation is distributed among mobiles

44 Z. Li, 2012

mobiles– The solution works well in convergence

Summary of the Multi-Access Video Networking

• The Video Adaptation + Resource Pricing Framework– For multimedia networking problems that can be expressed as a sum

utility (concave) maximization and resource sum constraint(s), this resource pricing with local surplus maximization framwork works.

» E.g, the game traffic shaping work, T-MM, 2008.

– When we have multiple sum resource constraints, e.g, over links in the network, then we have TCP like solution. Eg ,out T-MM 2009 work on CAF

45 Z. Li, 2012

CAF– When constraints can’t be expressed as a simple sum, then need more

advanced solutions like auction, and other game theoretical approaches.

Light Computational Complexity Solution

• Video Cache Pruning Solution– Reduced complexity solution– Provision extra storage and computing resource at base station controller

– Mixed transcoding / packet pruning solution to balance computational complexity

– Offline or online frame drop QoE metric computing:

50

60

46 Z. Li, 2012

20 40 60 80 100 120 1400

10

20

30

40

50

dist

ortio

n

20 40 60 80 100 120 1400

10

20

30

40

50

60

dist

ortio

n

frame k

Light Computational Complexity Solution

• Frame drop distortion gradient priority algorithm– Provision extra storage and computing resource at base station controller

– Mixed transcoding / packet pruning solution to balance computational complexity

– Offline or online frame drop QoE metric computing:

sorted utility gradient

Algorithm 2. Video Queue Prunning

1. Compute Gradient for all n video queues of m frames in the

buffer

47 Z. Li, 2012

05

1015

2025

30

0

2

4

6

80

5

10

15

20

25

frame video seq

dis

tort

ion/

rate

2. FOR k=1:n

a. Find the minimum gradient frame drop for user k,

j

j

mjk r

dg

]..1[min∈

=

b. Select the minimum gradient among users, }{minarg* kk

gk =

c. Drop user k*’s min gradient frame (and associated

decoding dependent frames)

d. Check if the total reduction in rate meet the reque st or not.

Exit if okay.

3. END

Frame Pruning Demo

• Set up 10 mobile users playing back youtube video o f 200~500kbps range

• Simulating base station bottleneck at 10%~40% throu ghput reduction case

8

9

10

dienbienphu−2

dienbienphu−4

dienbienphu−5

48 Z. Li, 2012

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

1

2

3

4

5

6

7

rate reduction ratio

dis

tort

ion

foreman

i16

korolev−4

korolev−5

me109

Demo !

Related Journal Papers

• QoE metric and very low bit rate video:� Z. Li, A. K. Katsaggelos, G. Schuster and B. Gandhi, "Rate-Distortion Optimal Video

Summary Generation", IEEE Trans. on Image Processing, pp. 1550-1560, vol. 14, no. 10, October, 2005.

� Z. Li, G. Schuster, A. K. Katsaggelos, "MINMAX Optimal Video Summarization and Coding", special issue on Analysis & Understanding for Media Adaptation, IEEE Trans. on Circuits and System for Video Technology, pp. 1245-1256, vol. 15, no. 10, October, 2005.

• Wireless Video Networking:� Y. Yang, Z. Li, W. Shi, Y. Chen, and H. Xu, "Cross-Layer Optimization for State Update in

Mobile Gaming", IEEE Trans. on Multimedia, vol. 10(5), pp. 701-710, August, 2008.

49 Z. Li, 2012

Mobile Gaming", IEEE Trans. on Multimedia, vol. 10(5), pp. 701-710, August, 2008. � J. Huang, Z. Li, M. Chiang, and A. K. Katsaggelos, "Joint Source Adaptation and Resource

Allocation for Multi-User Wireless Video Streaming", IEEE Trans. on Circuits & System for Video Tech, vol. 18 (5), pp. 582-595, May, 2008.

� Z. Li, F. Zhai, and A. K. Katsaggelos, "Joint Video Summarization and Transmission Adaptation for Energy Efficient Wireless Streaming", EURASIP Journal on Advances in Signal Processing, special issue on Wireless Video, vol. 2008, May, 2008

� W. Ji, Z. Li, and Y.-Q. Chen, "Joint Source-Channel Coding and Optimization for Layered Video Broadcasting to Heterogeneous Devices", in press, IEEE Trans on Multimedia, 2012.

• Internet and P2P Video Networking:� Y. Li, Z. Li, M. Chiang and A. Robert Calderbank, "Content-Aware Distortion Fair Video

Streaming in Congested Networks", IEEE Trans. on Multimedia, 2009

Future Work

• In the future– Richer yet practical QoE metric for wireless video, characterizing both

signal quality and playback temporal quality.– Effective and flexible suite of signaling and protocols to address multi-

access wireless video for newer generation wireless network stacks like E-UTRA.

– Integration with modern streaming solutions like DASH, Silverlight.

• Collaborators:

50 Z. Li, 2012

• Collaborators:– Prof. Mung Chiang, Princeton University – Prof. Robert Calderbank, Princeton University– Prof. Jianwei Huang, Chinese University of Hong Kong

Questions on Video Networking Research

?…

51 Z. Li, 2012

…Thanks!