On Multiple Foreground Cosegmentation

1

On Multiple Foreground Cosegmentation

Gunhee Kim Eric P. Xing

School of Computer Science, Carnegie Mellon University

June 18, 2012

2

• Problem Statement • Algorithm

Overview Foreground Modeling Region Assignment

• Experiments• Conclusion

Outline

3

Image Cosegmentation

High-level signal: recurring objects in multiple images

Jointly segment multiple images into K foregrounds and background

• [R06] Rother et al. CVPR2006 • [H09] Hochbaum and Singh, ICCV2009• [J10,J12] Joulin et al, CVPR2010, CVPR2012• [B11] Batra et al, IJCV 2011• [M11] Mukherjee et al, CVPR 2011• [V10,V11] Vincente et al, ECCV 2010, CVPR2011• [K11] Kim et al, ICCV 2011.

[R06] [J10]

[B11]

[K11]

4

Popular Cosegmentation Datasets

CMU-Cornell iCoseg Dataset [Batra et al. IJCV11]

MSRC Dataset [Winn et al. ICCV05]

Synthesized Dataset [Rhemann et al. CVPR09]

5

Popular Cosegmentation Datasets

CMU-Cornell iCoseg Dataset [Batra et al. IJCV11]

MSRC Dataset [Winn et al. ICCV05]

Synthesized Dataset [Rhemann et al. CVPR09]

Input images are carefully prepared by humanso that the objects of interest are salient enough in every single image.

6

General Users’ Photo Sets

A part of Apple+picking photo stream from Flickr

• A series of photos are taken for a specific moment• The number of foregrounds (ie. subjects of interest) are finite

Follow an ordinary user’s photo-taking pattern.

Girl in red (R)Girl in blue (G)Baby (B)Apple bucket (A)(R, G, A) (B, A) (R, A) (G, A)

(R, G, B, A) (R, G, A) (R, G, B, A) (R, A)

• Each image contains an unknown subset of foregrounds

7

General Users’ Photo Sets

A part of Apple+picking photo stream from Flickr

• A series of photos are taken for a specific moment• The number of foregrounds (ie. subjects of interest) are finite

Follows an ordinary users’ photo-taking pattern.

Girl in red (R)Girl in blue (G)Baby (B)Apple bucket (A)(R, G, A) (B, A) (R, A) (G, A)

(R, G, B, A) (R, G, A) (R, G, B, A) (R, A)

• Each image contains an unknown subset of foregrounds

Has NOT yet explicitly addressed

8

Problem StatementMultiple Foreground Cosegmentation

• Optionally, a user may assign some examples of foregrounds.

• Each image contains an unknown subset of foregrounds

Given an image set I, K foregrounds of interestSegment each image into regions of {F1,…,FK+1} (FK+1=B)

Girl in red (R)Girl in blue (G)Baby (B)Apple bucket (A)(R, G, A) (B, A) (R, A) (G, A)

(R, G, B, A) (R, G, A) (R, G, B, A) (R, A)

9

• Problem Statement • Algorithm

Overview Foreground Modeling Region Assignment

• Experiments• Conclusion

Outline

10

Overview of Algorithm

Iteratively solve

ForegroundModeling

RegionAssignment

• Learn appearance models of K+1 foregrounds (FGs)

• Performed in each image separately• Allocate the regions of image into one of K+1 FGs

Initialization

SupervisedAssigned by a user

UnsupervisedApply diversity ranking of [Kim et al. ICCV11]

Iterate until convergence

[Kim&Toralba NIPS09][Rother et al. SIGGRAPH04]

✔

11

• Problem Statement • Algorithm

Overview Foreground Modeling Region Assignment

• Experiments• Conclusion

Outline

12

Foreground (FG) Model

Definition of k-th FG model

Any region classifiers or their combination

Value Assignment (Testing) Foreground learning (Training)

Baby FG model

Baby FG model

A parametric function

Given any region S, return its value (score) of FG k

13

Foreground (FG) Model

Definition of k-th FG model

A parametric function

Given any region S, return its value (score) to FG k

Any region classifiers or their combination

Gaussian Mixture Model (GMM)

Spatial Pyramid + linear SVM (SPM)

• One of most popular in cosegmentation literatures

• One of most popular in image classification

• RGB colors • Gray/HSV SIFT

14

• Problem Statement • Proposed Algorithm

Foreground Modeling Region Assignment

• Experiments• Conclusion

Outline

15

Region Assignment

Given learned (or initialized) FG Models, RA is individually performed in each image

Cow FG Person FG

Background

Oversegment

16

Region Assignment

Given learned (or initialized) FG Models, RA is individually performed in each image

Cow FG Person FG

Background

Region Assignment

17

A Naïve Region Assignment

A naïve way: Assign each segment to the FG whose value is the highest

Cow FG Person FG

BackgroundCow BG BG BG Cow

However, it will NOT work !

18

Why does not A Naïve RA work?

Most naïve way: Assign each segment to FG whose value is the highest

Cow FG Person FGs1

s2

Cow won!

Person won!

My value of {s1,s2} is 18.

{s1,s2}My value of s2 is 5.

My value of s1 is 10.

My value of {s1,s2} is 35.

My value of s2 is 20.

My value of s1 is 7.

19

Why does not A Naïve RA work?

Most naïve way: Assign each segment to FG whose value is the highest

Cow FG Person FGs1

s2

Cow won!

Person won!

My value of {s1,s2} is 18.

{s1,s2}My value of s2 is 5.

My value of s1 is 10.

My value of {s1,s2} is 35.

My value of s2 is 20.

My value of s1 is 7.

Value of {s1,s2} to person FG

Value of s1 to Cow FG

Value of s2 to person FG +>

Have to evaluate the combinations of segments

20

Region Assignment by Combinatorial Auction

Cow FG Person FG

BackgroundItems to sell

Bidder 1 (or buyer)

Bidder 2

Bidder 3

• Each FG is allowed to bid packages of items with their own values.

• Distribute the segments to maximize the values.

21

Region Assignment as Combinatorial Auction

Assign the segments to FGs to maximize the overall values.

Winner determination problem (ie. Welfare maximization)

Unfortunately, NP-complete and Inapproximable

Feasibility: Each segment cannot be assigned more than once

[Cramton et al. 2005].

2 |Si|

subsets

22

✔

Next Goal: Tractable Solution to WDP

There are two different ways…

1. Constraints on value functions

2. Constraints on generating bidding packages

Allow any combinations of FG models (ie. Region classifiers)

• If value functions are submodular or subadditive, …

[Felzenszwalb et al.2008]

23

Procedure of Region Assignment

1. Each FG creates a set of foreground candidates

3. Solve WDP by choosing feasible FG candidates.

Follow a general combinatorial auction scenario.

: n FG candidates by FG k

: a bundle of segments : its value: FG ID

• In this step, each FG does not care their winning chances.

2. Finally,

24

Assumption for FG Candidates

A FG instance = a sub-tree of Gi

A FG instance in an image = a set of adjacent segments.

Any FG candidate = a sub-tree of Gi

25

Generating FG Candidatesby Beam Search

• Beam width D• Computation time: O(D|Si|2) • Number of FG candidates per FG: |Bi

k| = O(D|Si|)

For each size of candidates, we keep only D high valued ones.

2. Ot all possible subgraphs by adding

a single edge to elements of O.

1. O every single segment

3. Compute values vo vk(o) for all and keep only top D high valued ones to O.

4. Iterate (|Si|-1) times.

For each foreground k,

26

Example of Candidate Set

19.74 39.42 29.30

18.22 47.0639.47

39.6119.85 19.88

18.78

16.35

59.49

Back-ground

Apple bucket

Baby

27

Solving WDP

19.74 39.42 29.30

18.22 47.0639.47

39.6119.85 19.88

18.78

16.35

59.49

Back-ground

Apple bucket

Baby Solve WDP !

The number of FG candidates is (K+1)D|Si|.

Even a faster algorithm…

By search in polynomial time

28

Solving WDP

[Theorem] Dynamic program can solve WDP in O(|Bi||Si|)worst time if every candidate in Bi can be represented by a connected subgraph of a tree Ti

*.[Sandholm et al.2003]

Each FG candidate is a tree, but the aggregation is not.

19.74 39.42 29.30

18.22 47.0639.47

39.6119.85 19.88

18.78

16.35

59.49

Back-ground

Apple bucket

Baby

Candidate 2

Candidate 1

29

Inferring the Most Probable Tree from FG Candidate Set

19.74 39.42 29.30

18.22 47.0639.47

39.6119.85 19.88

18.78

16.35

59.49

Back-ground

Apple bucket

Baby

Solve the following MLE solution

: all possible spanning trees

where

w2 = 5Candidate 2

Candidate 1w1 = 20

Reject the bids that are not a subtree of

30

Inferring the Most Probable Tree from Candidate Set

19.74 39.42 29.30

18.22 47.0639.47

39.6119.85 19.88

18.78

16.35

59.49

Back-ground

Apple bucket

Baby

Solve the following MLE solution

: all possible spanning trees

where

MLE solution = MST by Kruskal’s algorithm in O(|Bi||Si|2)

Almost identical to Chow-Liu tree structure learning

31

Finally, Solve WDP

[Theorem] Dynamic program can solve WDP in O(|Bi||Si|)worst time if every candidate in Bi can be represented by a connected subgraph of a tree Ti

*.[Sandholm et al.2003]

optimal assignment segmentation

CABOB algorithm [Sandholm et al.2005]

32

• Problem Statement • Proposed Algorithm

Foreground Modeling Region Assignment

• Experiments• Conclusion

Outline

33

Two Experiments

FlickrMFC dataset

ImageNet dataset

• Goal: To achieve multiple foreground cosegmentation

• New benchmark dataset (14 groups, 20 images, fully-labeled)

ex. Cow group: {person, car, cow (brown, black)}

• Goal: Scalability

ex. green lizard

34

Quantitative Evaluation

Segmentation accuracies

Metric MFC-S: (supervised) our methodMFC-U: (unsupervised) our methodCOS : Submodular optimization [Kim et al. ICCV11]DC: Discriminative clustering [Joulin. CVPR10]LDA: LDA-based localization [Russell et al. CVPR06]MNcut: Normalized cuts [Cour et al. CVPR05]

35

Cosegmentation Examples

FlickrMFC

ImageNet

Australian terrier Lion

36

• Problem Statement • Proposed Algorithm

Foreground Modeling Region Assignment

• Experiments• Conclusion

Outline

37

Conclusion

• Each image contains a unknown subset of foregrounds

• Web-oriented applications

• Code and FlickrMFC dataset

will be available in this month!

• Fast and distributable (ex. Linear with M,K, polynomial in |Si|)

• Can be used for other tasks (ex.detection) beyond segmentation.

Multiple foreground cosegmentation

Combinatorial auction-based region assignment

38

Take-Home Message

Combinatorial optimization for Image Segmentation!

• New region descriptors & classifiers every year

• Multiple fast machines are available.

May avoid difficult ML algorithms or high-order models

Similar line of thought: Our ICCV 2011 paperSubmodular optimizationSimply enumerate the cases

(or hypotheses) not in a brute-force but in a smart way.