Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering

Comparison between Blur Transfer and Blur Re-

Generation in Depth Image Based Rendering

Norishige Fukushima†, Naoki Kodera†, Yutaka Ishibashi†, Masayuki Tanimoto‡

July 2-4, 2014 Budapest, Hungary 3DTV-CON 2014

†Graduate School of Engineering, Nagoya Institute of Technology,Japan

‡Nagoya Industrial Science Research Institute,Japan

Outline

Background Related Works View Synthesis Methods

Blur erasing type Blur re-generation type Blur transfer type

Experimental Results Conclusion and Future Works

Free Viewpoint

Image

Depth Image Based

Rendering

Original Image Depth Map

Background (1/3)

Background (2/3)

Input view and

depth map

3D warping

Hole filling

Free viewpoint image

Background (2/3)

Input view and

depth map

3D warping

Hole filling

Free viewpoint image

Background (3/3)

Blurred region

FB

Background (3/3)

Blurred region

FB

3D warp to left

Split

Background (3/3)

Blurred region

FB

3D warp to left

Split

Filled

Background (3/3)

Blurred region

FB

3D warp to left

Split

Filled

Boundary regions are degraded.

View synthesis method

3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type

Improved blur transfer method

Blur erasing type Blur re-generation type Blur transfer type

Improved blur transfer method (proposed method)

View synthesis method

3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type

Improved blur transfer method

Blur erasing type Blur re-generation type Blur transfer type

Improved blur transfer method (proposed method)

Blur erasing type

3D warp

Erase

Artifact

Blur erasing type

3D warp

Erase

Filled

Blurs are broken.

Artifact

View synthesis method

3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type

Improved blur transfer method

Blur erasing type Blur re-generation type Blur transfer type

Improved blur transfer method (proposed method)

Blur re-generation type

Generating blurred region using alpha matting

Splitting input image into three images→ foreground, background and alpha mask

Alpha blending three images after DIBR

Input image

Split

Foreground

Alpha mask

Background

Blur re-generation type

Fore

Alpha

Back

3D warp blendMatting

Blur re-generation

View synthesis method

3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type

Improved blur transfer method (proposed method)

Blur transfer type

Dilation

Color image

Depth map

Blur transfer type

Dilation

Color image

Depth map

Foreground depth value can cover almost fuzzy/blurred region.

Blur transfer type

Blur keep

3D warping

Blur transfer type

3D warping

interpolating

Blur keep

Canny and Gaussian Filtering

Dilation

DIBR

Blur transfer type

Improved blur transfer method (proposed method)

Adding a simple process

Improved blur transfer method

(proposed method)

Improved blur transfer method (proposed method)

Generating mask by canny filter Smoothing masked region by Gaussian

filterGenerate

mask

Smooth

Masked region

Comparison

Blur erasing type × Blurs are broken. ◎ Fastest

Blur re-generation type ○ Blurs are reconstructed. ×Slowest

Blur transfer type ○ Blurs are kept. ○ Faster

For better boundary treatment, we compare blur re-generation type with blur transfer type.

Experimental Results (1/6)

Evaluating PSNR of these methods Basic type [1] Blur re-generation type [2] Blur transfer type [3] Improved blur transfer method (proposed

method)

: TeddyH (950×750)

[1]: Y. Mori et al., Signal Process. Image Commu., vol. 24, no. 1, pp. 65-72, Jan. 2009. [2]: N. Kodera et al., IEEE VCIP 2013. [3]: X. Xu et al., IEEE ICASSP 2012.

: Bowling1 (671×555)

: Reindeer (671×555) Input depth map is ground truth

Experimental Results (2/6) PSNR

BasicRe-

generation

Transfer Proposed

average 35.62 38.85 37.93 38.54

TeddyH 32.74 35.03 34.97 35.13

Reindeer 34.37 38.14 36.75 38.15

Bowling1 34.42 39.72 35.46 35.93

. . . . .

. . . . .

. . . . .

. . . .

(dB)

#Average: using 30 data set

Experimental Results (3/6)

Basic

TeddyH: 950 x 750

Re-generation Proposed

Proposed method can soften contour artifacts, look like re-generation method.

Experimental Results (4/6)

Reindeer: 671 x 555

Re-generation Proposed

In proposed method, blurs around object boundary blend background color. Proposed method dilates depth value to

the unnecessary region.

Experimental Results (5/6)

Bowling1: 671 x 555

Re-generation Proposed Proposed method interpolates using

foreground or mixed color. Depth maps are not dilated enough.

Experimental Results (6/6)

Computational cost Proposed method ≒ Basic method (15ms) +

4ms 2 dilations for input depth map < 1ms Canny filter for depth map on the synthesized image

< 3ms Gaussian filter with a small kernel < 1ms

Blur re-generation type > 100ms Matting ≒ 55ms 3 times DIBR ≒ 45ms Blending < 2ms

Conclusions Comparing 3 types of DIBR method in blur

treatment. Proposed method improves subjective quality at object

boundaries, and objectively has 0.61dB improvement. This is the second best (0.31dB lower than the blur re-

generation). Proposed method reaches the state of the arts of blur

regeneration method, and computational cost is about x5 effective.

Proposed (19 ms) vs Blur re-generation (>100 ms)Future Work Investigating effect of proposed method using

estimated depth maps. Considering effect of coding distortions.