Silhouette Segmentation in Multiple Views

Silhouette Segmentation in Multiple Views

Wonwoo Lee, Woontack Woo, and Edmond Boyer

PAMI, VOL. 33, NO. 7, JULY 2011

Donguk Seoseodonguk@islab.ulsan.ac.kr

2012.10.13

2Intelligent Systems

Lab.

IntroductionExtraction of consistent foreground regions from multi views without a priori knowledge of the back-groundTwo assumptions for proposed method

The region of interest appears entirely in all images.Background colors are consistent in each image.

<Approach outline>

3Intelligent Systems

Lab.

Probabilistic modelVariables and their dependencies

: a color image map: a binary silhouette map: the prior knowledge about the model: the foreground occupancy: the background colors: th image: a pixel located at in an image: the color value of the pixel in the th image

<The variables in different views><Dependency graph of the image (Bayesian network)>

4Intelligent Systems

Lab.

Joint probabilityThe joint probability of all the variables

, , and : the prior probabilities of the scene, the foreground, and the background (uniform distribution): the silhouette likelihood that determines how likely is a silhouette given the foreground shape. (spatial consistency): the image likelihood term that models the relationship between the image observation. (colors and the background informa-tion)

Independent of background colors and foreground shape

| |

| , | , ,

Pr S,F,B,I,t Pr Pr B Pr F

Pr S F Pr I B S

(1)

,

,

| , | ,

| , , | , ,

ix x

i x

i i ix x x

i x

Pr S F Pr S F

Pr I B S Pr I B S

5Intelligent Systems

Lab.

Spatial consistency term(1/4): the probability of a silhouette S knowing the fore-ground shape FA spatial consistency term

To evaluate the silhouette consistency between viewpoints

Using the silhouette calibration ratioThe definition of silhouette set

Using a visual hull which is the maximal volume consistent with all silhouettes

| , | ,i i j ix xPr S F Pr S S

6Intelligent Systems

Lab.

Spatial consistency term(2/4)

<The silhouette consistencies of pixels in image >

7Intelligent Systems

Lab.

Spatial consistency term(3/4)The silhouette calibration ratio at pixel

A discrete measure based on the intersections be-tween the viewing ray at and the viewing cones from other viewpoints

: a normal distribution: the highest consistency value

, : the number of views: a normalization factor: controls how influences the silhouette consistency term

Using a value of 0.7 for

2 2/1 mxC C

xR ec

(2)

8Intelligent Systems

Lab.

Spatial consistency term(4/4)Spatial consistency term at given pixel location

: a uniform distributionThe silhouette information at that pixel

0: background1: foreground

, 0| ,

1

ii j i b xx j i

x x

P if SPr S S

R if S

(3)

9Intelligent Systems

Lab.

Image likelihood termThe image likelihood term

Similarity between a pixel color and the background infor-mation (the background color model at that location)

: the statistical model of the background colorsk-component Gaussian mixture model(GMM)

: the normal distribution with mean vector and covariance matrix

: controls the threshold between foreground and back-ground assignments and ranges from 0 to 1 (uniform distribution)

, 0| , ,

1

i iB x xi i i

x x j if x

H I if SPr I B S

P if S

(4)

| ,i iB x k x k k

k

H I w N I m (5)

10Intelligent Systems

Lab.

Inference of the silhouettesThe probability of the silhouette at pixel

0,1

0,1

| , , ,

, , , ,

, , , ,

| , | , ,

| , | , ,

ix

ix

i j i i ix x

i j i i ix x

i j i i ix xS

i j i i i ix x x

i j i i i ix x xS

Pr S S B I

Pr S S B I

Pr S S B I

Pr S S Pr I B S

Pr S S Pr I B S

(6)

11Intelligent Systems

Lab.

Iterative silhouette estimationTwo assumptions

(1) Any foreground element has an appearance dif -ferent from the background in most images so that color segmentation positively detects the element in most images.

(2) The region of interest appears entirely in all of the images considered.

Iterative opitmization(1) Silhouettes are estimated using foreground and

background models. (spatial and color consis-tencies)

(2) These model are updated with the new silhou-ettes.

12Intelligent Systems

Lab.

InitializationForeground scene: observed by all cameras

Belongs to the 3D space region that is visible from all cameras

The background color model

13Intelligent Systems

Lab.

Iterative optimization via Graph cut(1/2)Iteration

(1) Estimate each silhouette using (6) with the cur-rent background models and the other current silhouettes .

(2) Update each with pixels outside the current .

For the first stepTo decide for the pixel labeling into foreground or background in each image (from equation (6))

Graph-based approaches which account for additional spatial coherence in the image

14Intelligent Systems

Lab.

Iterative optimization via Graph cut(2/2)Minimization of energy of the pixel assignment in

image

: the data term that measures how good pixel label is with respect to the image observation.

: the smoothness term that favors consistent labeling in homogeneous region: the set or neighboring pixel pairs in image based on 8-connectivity

: the euclidean distance

,

| , ,

| , , ,i i

i ix y

i j i i it

i j i i i i id x x s x y

x I x y NS S

E S S B I

E S S B I E I I

(7)

| , , | , , ,i j i i i i j i i id x x x xE S S B I Pr S S B I

1,

1 ,i i

s x y i ix y

E I ID I I

15Intelligent Systems

Lab.

Silhouette refinement

: color model for foreground: Gaussian Mixture Model (GMM)

, 0

| , ,1

i iB x xi i i

x x i j iF x x

H I if SPr I B S

H I if S

(8)

(a) Input image. (b) Silhouette after the iterative optimization(c) Silhouette after refinements

16Intelligent Systems

Lab.

Experimental resultsSynthetic data

Kung-fu girl data set: http://www.mpi-inf.mpg.de/departments/irg3/kungfu/

17Intelligent Systems

Lab.

Kung-fu girl data (1/2)Segmentation results with different numbers of views (top row) accounting for spatial consistencies (bottom row)

Only using background color consistency

Two views four views Six viewsOne view

18Intelligent Systems

Lab.

Kung-fu girl data (2/2)Proposed method

19Intelligent Systems

Lab.

Real data(1/7)Basic calibrationGPU-based SIFT

<Segmentation results with the Dancer data (eight views) >

20Intelligent Systems

Lab.

Real data(2/7)

<Segmentation results with single-object scenes>

(10 views)

(12 views)

(5 views)

(12 views)

(8 views)

(6 views)

(8 views)

21Intelligent Systems

Lab.

Real data(3/7)Quantitative evaluation

: the number of pixels in a set: the label set a pixel (a: the labeling F or B, b: the ground truth label)

Table 1. Silhouette extraction performance measurements

FF

F FF B

BF

F BF F

N WHit Rate

N W N W

N WFalse Alarm Rate

N W N W

(9)

22Intelligent Systems

Lab.

Real data(4/7)

(a) Only one object is spatially consistent. (6 views)

(b) All three objects are spatially consistent. (6 views)

23Intelligent Systems

Lab.

Real data(5/7)<Convergence of the extracted silhouettes: the average false alarm rates at each iteration>

<Silhouette extraction with different >

24Intelligent Systems

Lab.

Real data(6/7)

<Silhouette extraction for a different number of views>

25Intelligent Systems

Lab.

Real data(7/7)<Silhouette extraction in the presence of noise>

26Intelligent Systems

Lab.

ConclusionsA novel method for extracting spatially consistent silhouettes of foreground objects from several view-points

Using spatial consistency and color consistency constraints in order to identify silhouettes with unknown backgrounds

The assumptionForeground objects are seen by all images and they present color differences with the back-ground regions.

27Intelligent Systems

Lab.

Thank you!!!

28Intelligent Systems

Lab.

Silhouette calibration ratioThe silhouette calibration ratio

: an interval along ray where image contributes rays and images : the number of image contributing inside that in-terval

1 max1 i

r

ir

wr i

C N wm n

29Intelligent Systems

Lab.

Visual hull

30Intelligent Systems

Lab.

Epipolar geometry3D point

Center of projection of camera

Projection of point X

Epipole

Epipolar plane

Epipolarline