Distributed Video System realized on mobile device with efficient Feedback channel

1

Distributed Video System realized on mobile device with efficient Feedback channel分散式影像編碼在手機上的實現與有效率的回饋通道Chen, chun-yuan 陳群元Advisor: Prof. Wu, Ja-Ling 吳家麟教授2012/7/9

2

Outline Motivation and introduction DVC architecture overview DVC architecture with mobile device Proposed algorithm for efficient feedback

channel Experimental Results Conclusion Future work

3

Motivation Video communication is essential in Mobile

device Decrease the coding time and battery

consuming on mobile.

4

Introduction For Mobile device video codec

Conventional video coding (ex:MPEG-4 H.264) Heavy weight encoder, light weight decoder.

Distributed Video Coding Light weight encoder, heavy weight decoder.

5

Conventional video codec

Encoding Decoding

6Tse-Chung Su, Yun-Chung Shen , and Ja-Ling Wu , “Real-time decoding for LDPC based distributed video coding,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2011 .

DVC to H.264 Transcoder

Clouding server:DVC to H.264 transcoder

DVC encoder H.264 decoder

Feedback

channel

Parity bits

Encoded sequence

LDPCA decoding is the bottleneck of the overall codec system.

7

DVC codec baseline: DISPAC+DVC transcoder

1.Encoder2.Decoder

Tse-Chung Su. Yun-Chung Shen. and Ja-Ling Wu. 2011. Real-time Decoding for LDPC Based Distributed Video Coding. National Taiwan University

without an actual feedback channel !

8

Time complexity analysis without feedback channel

DVC Decoding without Feedback channel Side-Information generation LDPCA

time ratio

SI generationLDPCA Decetc.

31%

Total decoding time: 17.10 sec

Tse-Chung Su, Yun-Chung Shen , and Ja-Ling Wu , “Real-time decoding for LDPC based distributed video coding,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2011 .

9

Time complexity analysis with feedback channel

DVC Decoding Side-Information generation Send-Request at Feedback channel LDPCA

time ratio

SI generationLDPCA DecFeedback channeletc.

1%

96%Bottle neck!

Total decoding time: 446.83 sec

T. V. Lakshman and U. Madhow, “The performance of TCP/IP for networks with high bandwidth-delay products and random loss, IEEE/ACM Trans. Networking, vol. 6, pp. 336–350, June 1997.

10

Outline Motivation and introduction DVC architecture Overview DVC architecture with mobile device Proposed algorithm for efficient feedback


11

Distributed Video Codec

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder

Buffer LDPCA Decoder

Key FramesConventional

IntraframeEncoder

Conventional Intraframe

Decoder

SI generation&refinement

DCT

SI ReconstructionBitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise

Modeling

IDCT&Deblocking filter

Decoded WZ Frames

Decoded Key Frames

WZFrame

WZFrame

WZFrame

GOP size 4

KeyFrame

KeyFrame

LDPC decoding

1 0 1

syndrome bits

Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm. IEEE Trans. Inform. Theory

a b c

1 2 3 4 5 6 7

Vertical processing

Horizontal processing

a b c d e f g

a1 b1 c1 d1 e1 f1 g1

a25 b25 c25 d25 e25 f25 g25

Side Information(real number)+ 0 - 1

decode outputhard decision

LDPC Accumulate (LDPCA) codes

13

Rate adaptivity

D. Varodayan et al., "Rate-adaptive codes for distributed source coding," EURASIP Signal Processing Journal, Special Section on Distributed Source Coding, 2006

65 L

DPC

cod

es

15



Distributed Video Codec with mobile

16

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Decoder


DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise Modeling


Decoded WZ Frames

Decoded Key Frames

Transcode Decoded Video sequenceWith specific form

Video deliver

Android java coding


17

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Decoder


DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel



Decoded WZ Frames

Decoded Key Frames


Video deliver

18

Key frame encoding on Mobile

Previous ver. use JM9.5 Transcoder ref x264 on PC

x264 lib

Cross compilerKeyframe encoding

x264 static

library


19

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Decoder


DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel



Decoded WZ Frames

Decoded Key Frames


Video deliver

20

WZ coding with NDK

Native code

DVC encoder C/C++ code with NDK rule

Jni Makefile


21

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Decoder


DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel



Decoded WZ Frames

Decoded Key Frames


Video deliver

22

Network connection Create network connection to connect

remote server. TCP socket in C

Clouding server:DVC to H.264

transcoder

23

ACK at Feedback channelTCP socket

…

syndromes

Send syndrome

Request

syndromessyndromes

syndromes

24

Decoding complexity onFeedback channel

Packet header occupy large part of network bandwidth. 48 bits syndromes vs. 24 bytes header

Network latency in communication at Feedback channel About 20 thousands times requests for 10 sec

sequence.

25

Outline Motivation and introduction DVC architecture with mobile device DVC architecture overview Proposed algorithm for efficient feedback


26

Efficient feedback channel we propose two methods to decrease time

consuming at feedback channel. Estimate the syndrome size per bitplane Estimate the syndrome size per WZ frame

Number of requests per bitplane per band

1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000 bit rate

bit rate

DC AC1 AC2 AC3 AC4 AC5 AC6 AC7 AC8 AC9AC10AC11

AC12AC13

AC14

Bitplane Number


1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000 bit rate

bit rate


AC12AC13

AC14

Bitplane Number


1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000

bit rateestimated bit rate


AC12AC13

AC14

Bitplane Number

30

Efficient feedback channel we propose two methods to decrease time

consuming at feedback channel. Estimate the syndrome size per bitplane Estimate the syndrome size per WZ frame

Average number of requests per bitplane

1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000 bit rate

bit rate


AC12AC13

AC14

Bitplane Number

32

Estimate the syndromes size per WZ frame

Estimate syndroms size by correspond WZ frame in previous GOP

Ex. GOP 4

Key frame

WZ frame

WZ frame

WZ frame

First GOP

Key frame

Key frame

WZ frame

WZ frame

WZ frame

Second GOP

33



Test condition 12 CPU, 24 processor

Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GPU: Tesla M2050 Mobile device: HTC sensation

Test sequences :

QCIF, 15Hz, all frames GOP Size 2, 4 and 8 Only luminance component is used

Test materials

Soccer Foreman Coastguard Hall Monitor

Motion: High Low

36

Decoding timeMethod\video foreman soccer coastguard hall

With feedback

channel

446.83 sec 419.77 sec 482.40 sec 417.29 sec

Estimate per

bitplane

237.56 sec 237.56 sec

271.61 sec

305.15 sec

Estimate per WZ

frame

104.66 sec 73.28 sec 109.27 sec 78.96 sec

37

Request amount

with feedback channel

estimate per bitplane

estimate per WZ frame

0

5000

10000

15000

20000

25000

30000

35000

foremansoccercoastguardhall

38

Speed up ratio

estimate per bitplane estimate per WZ frame0

1

2

3

4

5

6

7

Speed up ratio(with feedback channel=1)

foremansoccercoastguardhall

Methods

Spee

d u

p r

atio

39

Bitrate

with feedback channel



0100200300400500600700800

BitRate(kbps)

foremansoccercoastguidehall

methods

Bit

Rat

e(k

bp

s)

40

RD(Foreman)

41

RD(hall)

42



43

Conclusion We build a realistic DVC system based on

mobile and cloud. Propose algorithm for Feedback channel

which the previous work doesn’t discuss or improve.

Transcode the decoded sequence to mobile codec.

44



45

Future Work Take a more precise estimation of syndrome

size. Use motion information to estimate the

syndrome size. Build a reliable DVC codec which is loss

tolerant.

46

Thank You

47

streaming

WZFrame

WZFrame

WZFrame

GOP size 4

KeyFrame

KeyFrame

WZFrame

WZFrame

WZFrame

GOP size 4

KeyFrame

KeyFrame

48

WZFrame

WZFrame

WZFrame

GOP size 4

KeyFrame

KeyFrame

WZFrame

WZFrame

WZFrame

GOP size 4

KeyFrame

KeyFrame

50

To do Real time decoder Efficient intra coding Print MSG on Mobile

51

DVC decoding time Side Info Creation: 0.85 sec Overcomplete Transform: 2.35 sec Correlation Noise Modeling: 2.33 sec Conditional Bit Prob Compute: 0.60 sec Update Side Info: 7.14 sec Ldpca Decode: 123.34 sec Motion Learning: 1.79 sec Others: 1.80 sec

55


DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Encoder


DCT


Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise

Modeling


Decoded WZ Frames

Decoded Key Frames


56

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Encoder


DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel



Decoded WZ Frames

Decoded Key Frames


Video deliver

57

Video recodeWith mobile camera

58

ACK at Feedback channel

…

Send syndrome

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes for one bitplane

59


…

Request

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes for one bitplane Decoding one bitplane

with received syndromesAnd chack CRC

60


…

Send syndrome

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes for one bitplane

65

Contribution Realize DVC encoding part on Mobile device Implement Feedback channel Implement encoder’s component with x264 Propose an efficient Feedback channel Transcode the decoded sequence to mobile

codec

66

Outline Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

Quantization

Q8

2^7

2^6

2^6

2^5

2^5

2^5

2^4

2^4

2^4

2^4

2^3

2^3

2^3

2^2

2^2

63bits for one block

Q5

2^52^4

2^4

2^3

2^2

2^3 2^2

2^3 2^2 2^2

2^2 2^2

2^2

37bits for one block

72


DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Encoder


DCT


Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise

Modeling


Decoded WZ Frames

Decoded Key Frames

73


74

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Encoder


DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel



Decoded WZ Frames

Decoded Key Frames


Video deliver


75

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder



IntraframeEncoder


Encoder


DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel



Decoded WZ Frames

Decoded Key Frames


Video deliver

76

DVC Encoder Realization Implement DVC encoder on Mobile

Media recording Key frame encoding WZ frame encoding

Feedback channel transmission Network connection between mobile and server Transcode the result sequence

method Syndromes need (bits)

Request number

Cover rate(bitplanes num=130*63=8190)

Total decoding time(speed up)

LDPCA time(speed up)

BitRate PSNR

naive 2575608 2575608(+130*63* 不算server 來的request 只算 mobile 傳送syndrome)

13hr=46800 13hr(with feedback channel)

450.32 kbps 39.29

rearrangement

2575608 23983 406.20 sec 386.90 sec 450.32 kbps

estimator 3715344 3608 6734/8190=0.822

87.99 sec(4.61)

70.36 sec(5.498)

564.29 kbps

parallel 2825256 15736( 可以不用再送，只靠 pre-send 的bitplane 數=3249)

3249/8190=0.396

237.56 sec(1.709)

219.14 sec(1.765)

475.29 kbps


Request number




BitRate PSNR



450.32 kbps 39.06

rearrangement

1922376 20416 448.61 sec 413.36 sec 485.96 kbps

estimator 3902256 2419 7291/8190=0.89

100.20 sec 72.27 sec(5.719)

683.95 kbps

parallel 2825256

?( 卡在 frame2)

15736( 可以不用再送，只靠 pre-send 的bitplane 數=3249)

3249/8190=0.396

237.56 sec 219.14 sec(1.886)

475.29 kbps


Request number




BitRate PSNR



450.32 kbps

rearrangement

3586656 28721 431.45 sec 395.96 sec 479.39 kbps

estimator 3715344

?(60後死掉 ..)

3608 6734/8190=0.822

87.99 sec 70.36 sec(5.627)

564.29 kbps

parallel 3719712 21512( 可以不用再送，只靠pre-send 的bitplane 數=3249)

2572/8190=0.314

271.61 sec 244.18 sec(1.621)

492.70 kbps


Request number




BitRate PSNR



450.32 kbps 39.47

rearrangement

924696 17440 505.84 sec 477.45 sec 153.45 kbps

estimator 1385736 1852 7823/8946=0.874

109.75 sec 93.24 sec(5.12)

195.62 kbps

parallel 971376 10568( 可以不用再送，只靠pre-send 的bitplane 數=3249)

3367/8946=0.376

305.15 sec 285.59 sec(1.671)

157.72 kbps

formula (ESbt)WZn=(Sbt)WZ(n-GOPsize)

WZn means the WZ frame’s index n-GOPsize means the correspond WZ frame in

the previous GOP. bt is the bitplane index. (ESbt)WZn means the estimated syndromes bit for

the bt’th bitplane of WZn frame. (Sbt)WZ(n-GOPsize) means the syndromes bit for the

bt’th bitplane of WZ(n-GOPsize) frame

formula (ESacn

bt)WZn= ( Sdc bt-1 )WZn WZn means the n’th WZ frame. acn means the n’th AC band. bt means the bitplane index of this AC band. Which ESacn

bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band.

dc means the DC band. Sdc bt-1 means the syndromes bits for bt-1’th

bitplane in DC band.

83

Time complexity

time ratio


time ratio


95%

75%

84

bottleneck component of decoding process is always the LDPCA decoding

Yu-Shan Pai, Han-Ping Cheng, Yun-Chung Shen,and Ja-Ling Wu , “Fast decoding for LDPC based distributed video coding,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2010 .

Transcode to mobile codec Because DVC only encode luminance part in

YUV, so we got only Y for output. Our mobile device can only read mp4 and

3gp.

Y

U V

R G BFrame 1Frame 2Frame 3Frame4Frame 5Frame 6Frame 7Frame n.mp4

86

Correlation with LDPCA

SI quality

Residual

統計關係圖

0 100000 200000 300000 400000 500000 600000 7000000

50001000015000200002500030000350004000045000

Syndromes bits

syn

Residual value

Syn

dro

mes

bit

s

Documents

Distributed Video System realized on mobile device with efficient Feedback channel