Graph‐based Segmentation

Computer VisionCS 543 / ECE 549 

University of Illinois

Derek Hoiem

02/25/10

Last class

• Gestalt cues and principles of organization

• Mean‐shift segmentation– Good general‐purpose segmentation method 

– Generally useful clustering, tracking technique

• Watershed segmentation– Good for hierarchical segmentation

– Use in combination with boundary prediction

Today’s class

• Treating the image as a graph–Normalized cuts segmentation

–MRFs Graph cuts segmentation

• Recap

• Go over HW2 instructions

i

Images as graphs

• Fully‐connected graph– node for every pixel

– link between every pair of pixels, p,q

– similarity wij for each link

j

wijc

Source: Seitz

Similarity matrix

Increasing sigma

Segmentation by Graph Cuts

• Break Graph into Segments– Delete links that cross between segments– Easiest to break links that have low cost (low similarity)

• similar pixels should be in the same segments• dissimilar pixels should be in different segments

w

A B C

Source: Seitz

Cuts in a graph

• Link Cut– set of links whose removal makes a graph disconnected– cost of a cut:

A B

One idea: Find minimum cut• gives you a segmentation• fast algorithms exist for doing this

Source: Seitz

But min cut is not always the best cut...

Cuts in a graph

A B

Normalized Cut• a cut penalizes large segments• fix by normalizing for size of segments

• volume(A) = sum of costs of all edges that touch A

Source: Seitz

Recursive normalized cuts1. Given an image or image sequence, set up a weighted 

graph: G=(V, E)– Vertex for each pixel– Edge weight for nearby pairs of pixels 

2. Solve for eigenvectors with the smallest eigenvalues: (D − W)y = λDy

– Use the eigenvector with the second smallest eigenvalueto bipartition the graph

– Note: this is an approximation

4.   Recursively repartition the segmented parts if necessary

http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdfDetails:

Normalized cuts results

Normalized cuts: Pro and con• Pros

– Generic framework, can be used with many different features and affinity formulations

– Provides regular segments

• Cons– Need to chose number of segments

– High storage requirement and time complexity

– Bias towards partitioning into equal segments

• Usage– Use for oversegmentation when you want 

regular segments

Graph cuts segmentation

Markov Random Fields

∑∑∈

=edgesji

jii

i datayydataydataEnergy,

21 ),;,(),;(),;( θψθψθy

Node yi: pixel label

Edge: constrained pairs

Cost to assign a label to each pixel

Cost to assign a pair of labels to connected pixels

Markov Random Fields

• Example: “label smoothing” gridUnary potential

0 10 0 K1 K 0

Pairwise Potential

0: -logP(yi = 0 ; data)1: -logP(yi = 1 ; data)

∑∑∈

=edgesji

jii

i datayydataydataEnergy,

21 ),;,(),;(),;( θψθψθy

Solving MRFs with graph cuts

∑∑∈

=edgesji

jii

i datayydataydataEnergy,

21 ),;,(),;(),;( θψθψθy

Source (Label 0)

Sink (Label 1)

Cost to assign to 0

Cost to assign to 1

Cost to split nodes

Solving MRFs with graph cuts

∑∑∈

=edgesji

jii

i datayydataydataEnergy,

21 ),;,(),;(),;( θψθψθy

Source (Label 0)

Sink (Label 1)

Cost to assign to 0

Cost to assign to 1

Cost to split nodes

Grab cuts and graph cuts

User Input

Result

Magic Wand(198?)

Intelligent ScissorsMortensen and Barrett (1995)

GrabCut

Regions Boundary Regions & Boundary

Source: Rother

Colour Model

Gaussian Mixture Model (typically 5-8 components)

Foreground &Background

Background

Foreground

BackgroundG

R

G

RIterated graph cut

Source: Rother

Graph cutsBoykov and Jolly (2001)

ImageMin Cut

Cut: separating source and sink; Energy: collection of edges

Min Cut: Global minimal enegry in polynomial time

Foreground (source)

Background(sink)

Source: Rother

Graph cuts segmentation

1. Define graph – usually 4‐connected or 8‐connected

2. Define unary potentials– Color histogram or mixture of Gaussians for 

background and foreground

3. Define pairwise potentials

4. Apply graph cuts5. Return to 2, using current labels to compute 

foreground, background models

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧ −−

+= 2

2

21 2)()(

exp),(_σ

ycxckkyxpotentialedge

⎟⎟⎠

⎞⎜⎜⎝

⎛−=

));(());((

log)(_background

foreground

xcPxcP

xpotentialunaryθθ

Moderately straightforward examples

… GrabCut completes automaticallyGrabCut – Interactive Foreground Extraction 10

Difficult Examples

Camouflage & Low Contrast Harder CaseFine structure

Initial Rectangle

InitialResult

GrabCut – Interactive Foreground Extraction 11

Using graph cuts for recognition

TextonBoost (Shotton et al. 2009 IJCV)

Using graph cuts for recognition

TextonBoost (Shotton et al. 2009 IJCV)

Unary Potentials

Alpha Expansion Graph Cuts

Limits of graph cuts

• Associative: edge potentials penalize different labels

• If not associative, can sometimes clip potentials

• Approximate for multilabel– Alpha‐expansion or alpha‐beta swaps

Must satisfy

Graph cuts: Pros and Cons

• Pros– Very fast inference– Can incorporate recognition or high‐level priors– Applies to a wide range of problems (stereo, image labeling, recognition)

• Cons– Not always applicable (associative only)– Need unary terms (not used for generic segmentation)

• Use whenever applicable

Further reading and resources

• Normalized cuts and image segmentation (Shi and Malik)http://www.cs.berkeley.edu/~malik/papers/SM‐ncut.pdf

• N‐cut implementation http://www.seas.upenn.edu/~timothee/software/ncut/ncut.html

• Graph cuts– http://www.cs.cornell.edu/~rdz/graphcuts.html

– Classic paper: What Energy Functions can be Minimized via Graph Cuts? (Kolmogorov and Zabih, ECCV '02/PAMI '04)

Recap of Grouping and Fitting

Line detection and Hough transform• Canny edge detector =            smooth  derivative  thin threshold  link

• Generalized Hough transform = points vote for shape parameters

• Straight line detector =               canny + gradient orientations orientation binning  linking check for straightness

Robust fitting and registration

Key algorithm

• RANSAC

Clustering

Key algorithm

• Kmeans

EM and Mixture of Gaussians

Tutorials: http://www.cs.duke.edu/courses/spring04/cps196.1/.../EM/tomasiEM.pdfhttp://www‐clmc.usc.edu/~adsouza/notes/mix_gauss.pdf

Segmentation• Mean‐shift segmentation

– Flexible clustering method, good segmentation

• Watershed segmentation– Hierarchical segmentation from soft boundaries

• Normalized cuts– Produces regular regions

– Slow but good for oversegmentation

• MRFs with Graph Cut– Incorporates foreground/background/object 

model and prefers to cut at image boundaries

– Good for interactive segmentation or recognition

Next section: Recognition

• How to recognize– Specific object instances

– Faces

– Scenes

– Object categories

– Materials