{ Fast Disparity Estimation Using Spatio- temporal Correlation of Disparity Field for Multiview Video Coding Wei Zhu, Xiang Tian, Fan Zhou and Yaowu Chen

{

Fast Disparity Estimation Using Spatio-temporal Correlation of Disparity Field for Multiview Video Coding

Wei Zhu, Xiang Tian, Fan Zhou and Yaowu ChenIEEE TCE 2010

2

Introduction Analysis of Disparity Field in MVC Proposed Fast Disparity Estimation Algorithm Experimental Results Conclusion

Outline

3

Fig. 1. MVC coding schemes among views for different sequences.

(a) For “Ballroom” sequence with general 1D camera setup. (b) For “Akko&Kayo” sequence with 2D-array camera setup. (c) For “Flamenco2" sequence with 2D-cross camera setup.

Introduction

4

Motion Estimation Temporal motion is dependent on the movement of

objects. Only moving objects have motion displacements,

objects in background often have no motion displacement.

Disparity Estimation Inter-view disparity is dependent on the depth of

object and the camera setup. Objects with no motion displacements may also have

large disparity displacements.

Introduction

5

Analysis of Disparity Field in MVC

[17] X. M. Li, D. B. Zhao, S. W. Ma and W. Gao, “Fast disparity and motion estimation based on correlations for multiview video coding,” IEEE Trans. Consumer Electron., vol. 54, no. 4, pp. 2037-2044, Nov. 2008.

object depth

camera parameters

6

Disparity is related to the depth of objects Shallow depths large disparity Distant depths small disparity

Objects in MVC have large range of depths, the range of disparity is also large.

DE requires a large search range to find optimal disparity vector(DV).


7


Fig. 2. Histogram of horizontal disparity vectors between view S1 and view S0 for “Ballroom” sequence.

Size of DV vary a lot: Most of the DV are in [0,

24], background objects. Some DV are in [24, 72],

foreground objects.

Most of DV are in the positive.

The disparity direction

is only determined by the location of views.

8

Spatial direction correlation

Fig. 3. A field of original disparity vectors for the 123th frame of view S1 in “Ballroom” sequence. [14] Y. Kim, J. Kim, and K. Sohn, “Fast disparity and motion estimation for multi-view video coding,” IEEE Trans. Consumer Electron., vol. 53, no.2, pp.712-719, May 2007.


DVs are highly correlated with neighboring vectors in spatial direction.[14]

Some irregular DVs are obvious different with their neighbors, especially in homogenous regions.

9

Temporal direction correlation

Fig. 4. Histogram of temporal difference for horizontal disparity vectors between view S1 and view S0 in “Ballroom” sequence.


DVs in the temporal direction are also highly correlated.

10

The characteristics of disparity field in MVC: The size of DVs could vary a lot because of different depths

of objects. Due to the fixed camera setup, most of DVs have a consistent direction with the real disparities.

Some DVs deviate from the real disparities, DVs of previous coded should be filtered to be consistent with real disparities.

Since DVs have a high correlation with neighboring vectors in the temporal and spatial direction, the search center of DVs and preliminary DV can be predicted.


11

Basic idea: Select the search center by using the spatio-temporal correlation of disparity field, and to predict the search range adaptively according to the temporal variation of disparity field.

Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame.

Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field.

Part C: Prediction of Search Range Based on the Temporal Variation of Disparity Field.

Part D: Overall Proposed Algorithm.

Proposed Fast Disparity Estimation Algorithm

12

Basic idea: There is a high correlation of DVs in temporal direction. Use previous coded frames to predict current frame.

Problem: Some noisy DVs are not consistent with their real disparities, these vectors should be eliminated to obtain a smooth disparity field.

Solution: Because noisy DVs have irregular directions, check every DV direction first.

Requirement: GDV is calculated by averaging all DVs of block 16x16 in the previous coded frame, and its direction is used as the reference of the real disparity direction.

Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame

13

Step1: Using GDV, the DV of block 16x16 for each MB is regularized.


90 °

number of vectors in 1

14

Step2: To obtain a smooth variation in the spatial direction, is further filtered by a spatial median filter.

A field of the :


15

Step3: After obtaining a smoothed disparity field of the previous coded frame, a temporal prediction of the DV () is calculated for each MB in current frame.


encoding order of current frame

0.5 co-located

(Zero Disparity Vector)

16

Basic idea: The disparity vectors are highly correlated in spatio-temporal directions, so the neighboring disparity vectors in spatio-temporal directions are used to determine the search center for the current block.

Select candidates: The neighboring disparity vectors are selected as candidates of

the search center. Non-anchor frame: 5-9 are selected from the forward and backward

temporal reference frame. Anchor frame: 5-9 are selected from the previous coded anchor

frame.

Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field

17

: Considering the influence of homogenous regions. : , it is a smoothed prediction of the disparity vector from spatial

direction. : It is a temporal disparity predictor.

All of the available candidates for the current block are made up the set .

To prevent digression of DE, candidates of the search center are further limited around by a given maximum search range().


18

The candidate which has minimum Lagrangian cost is selected as the search center of the disparity vector ().

The average Manhattan distance between CDV and the optimal DV estimated by the full search algorithm.


It’s the smallest, only a smaller search range is needed.

19

Temporal variation(distance) of DV: |CDV – TDV| CDV: An approximation of the DV in current frame. TDV: A temporal prediction of the DV in previous

coded frame.

Basic idea: the distance is related the DV consistency between current frame and previous coded frame.

Small distance small search range Large distance large search range


20

Search range:


(larger)

1.3

12

6

Calculate search range

21

Part D: Overall Proposed Algorithm

𝐷𝑉 ’

𝑇 𝐷𝑉

𝐶𝐷𝑉

𝐷𝑉 𝑜𝑝𝑡𝑖𝑚𝑎𝑙

22

JMVC4.0 Testing Configuration

Only Inter16x16 mode for inter mode Use first two or three views of the sequences. View S1 was chosen for proposed algorithm, and

disable ME S0 was chosen as the reference view for

Flamenco2(2D-cross), S0 and S2 were chosen for others.

Comparing with full search algorithm and fast search algorithm.

Experimental Results

23

Rate-distortion performance


24

Performance comparison


25

In this paper, a fast DE algorithm is proposed to save the computational load of MVC.

Take into account the spatial-temporal correlation and the temporal variation of disparity.

Perform well on all test sequences. Compare with full search algorithm, achieve an

average 96% reduction of computational complexity, while RD performance remain the same.

Compare with fast search algorithm, achieve an average 43% reduction of computational complexity, while RD performance is improved.

Conclusion

Documents

{ Fast Disparity Estimation Using Spatio- temporal Correlation of Disparity Field for Multiview Video Coding Wei Zhu, Xiang Tian, Fan Zhou and Yaowu Chen