POSE–CUTSimultaneous Segmentation and 3D Pose

Estimation of Humans using Dynamic Graph Cuts

Mathieu Bray Pushmeet Kohli Philip H.S. Torr

Department of Computing

Oxford Brookes University

Objective

Image Segmentation Pose Estimate

[Images courtesy: M. Black, L. Sigal]

Outline

Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work

Outline

Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work

The Image Segmentation ProblemSegments

Image

Problem – MRF Formulation

Notation• Labelling x over the set of pixels• The observed pixel intensity values y (constitute data D)

Energy E (x) = - log Pr(x|D) + constant

Unary term• Likelihood based on colour

Pairwise terms• Prior• Contrast term

Find best labelling x* = arg min E(x)

MRF for Image Segmentation

D (pixels)

x (labels)

Image Plane

i

j

xi

xj Unary Potential

i(D|xi)

Pairwise Potential

ij(xi, xj)

xi = {segment1, …, segmentk} for instance {obj, bkg}

Can be solved using graph cuts

MRF for Image Segmentation

MAP SolutionPair-wise Terms

Contrast Term

IsingModel

Data (D) Unary likelihood

Unary likelihood

Maximum a-posteriori (MAP) solution x* =

MRF for Image Segmentation

Pair-wise Terms MAP SolutionUnary likelihoodData (D)

Unary likelihood

Contrast Term

Uniform Prior

Maximum-a-posteriori (MAP) solution x* =

Need for a human like segmentation

Outline

Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work

Shape-Priors and Segmentation

OBJ-CUT [Kumar et al., CVPR ’05]– Shape-Prior: Layered Pictorial Structure (LPS)– Learned exemplars for parts of the LPS model– Obtained impressive results

Layer 2Layer 1Spatial Layout

(Pairwise Configuration)

+ =

Shape-Priors and Segmentation

OBJ-CUT [Kumar et al., CVPR ’05]– Shape-Prior: Layered Pictorial Structure (LPS)– Learned exemplars for parts of the LPS model– Obtained impressive results

Shape-Prior Colour + ShapeUnary likelihoodcolour

Image

Problems in using shape priors

Intra-class variability• Need to learn an

enormous exemplar set• Infeasible for complex

subjects (Humans)

Multiple Aspects?

Inference of pose parameters

Do we really need accurate models?

Interactive Image Segmentation [Boykov & Jolly, ICCV’01]• Rough region cues sufficient • Segmentation boundary can be extracted from edges

additional segmentation

cues

user segmentation cues

Do we really need accurate models? Interactive Image Segmentation

• Rough region cues sufficient • Segmentation boundary can be extracted from edges

Rough Shape Prior - The Stickman Model

26 degrees of freedom• Can be rendered extremely efficiently• Over-comes problems of learning a huge exemplar set• Gives accurate segmentation results

Pose-specific MRF Formulation

D (pixels)

x (labels)

Image Plane

i

j

xi

xj Unary Potential

i(D|xi)

Pairwise Potential

ij(xi, xj)

(pose parameters)

Unary Potentiali(xi|)

Pose-specific MRFEnergy to be

minimizedUnary term

Shape prior

Pairwise potential

Potts model

distance transform

Pose-specific MRFEnergy to be

minimizedUnary term

Shape prior

Pairwise potential

Potts model

+ =

Shape Prior

MAP Solution

Colourlikelihood

Data (D) colour+shape

What is the shape prior?Energy to be

minimizedUnary term

Shape prior

Pairwise potential

Potts model

How to find the value of

ө?

Outline

Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work

Formulating the Pose Inference Problem

Formulating the Pose Inference Problem

Resolving ambiguity using multiple views

Pose specific Segmentation Energy

Outline

Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work

Solving the Minimization ProblemSolving the Minimization Problem

Minimize F(ө) using Powell Minimization

To solve:

Let F(ө) =

Computational Problem:

Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT..

Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

Dynamic Graph Cuts

PB SB

cheaperoperation

computationally

expensive operation

Simplerproblem

PB*

differencesbetweenA and B

A and Bsimilar

PA SA

solve

Dynamic Graph Cuts

20 msec

Simplerproblem

PB*

differencesbetweenA and B

A and Bsimilar

xasolve

xb400 msec

Outline

Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work

Segmentation Results

Colour +Smoothness

Colour + Smoothness+ Shape Prior

Only Colour

Image

[Images courtesy: M. Black, L. Sigal]

Segmentation Results - Accuracy

Information used

% of object pixels correctly

marked

Accuracy(% of pixels correctly

classified)

Colour 45.73 95.2

Colour + GMM 82.48 96.9

Colour + GMM + Shape

97.43 99.4

Segmentation + Pose inference

[Images courtesy: M. Black, L. Sigal]

Segmentation + Pose inference

[Images courtesy: Vicon]

Outline

Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work

Conclusions

• Efficient method for using shape priors for object-specific segmentation

• Efficient Inference of pose parameters using dynamic graph cuts

• Good segmentation results

• Pose inference- Needs further evaluation- Segmentation results could be used for silhouette intersection

Future Work

• Use dimensionality reduction to reduce the number of pose parameters.

- results in less number of pose parameteres to optimize- would speed up inference

• Use of features based on texture

• Appearance models for individual part of the articulated model (instead of using a single appearance model).

Thank You