Segmentation

Terminology• Segmentation, grouping, perceptual

organization: gathering features that belong together

• Fitting: associating a model with observed features

• Top-down segmentation: pixels belong together because they come from the same object

• Bottom-up segmentation: pixels belong together because they look similar

The goals of segmentation• Separate image into coherent “objects”

Berkeley segmentation database:http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/

image human segmentation

The goals of segmentation• Separate image into coherent “objects”

• Top-down or bottom-up process?• Supervised or unsupervised?

• Group together similar-looking pixels for efficiency of further processing

X. Ren and J. Malik. Learning a classification model for segmentation.ICCV 2003.

“superpixels”

The goals of segmentation• Separate image into coherent “objects”

• Top-down or bottom-up process?• Supervised or unsupervised?

• Group together similar-looking pixels for efficiency of further processing• Related to image compression• Measure of success is often application-dependent

Segmentation: Outline• Inspiration from psychology• Segmentation as clustering

• K-means• Mean shift

• Segmentation as partitioning• Graph-based segmentation, normalized cuts

• Integrating top-down and bottom-up segmentation for recognition

The Gestalt school• Grouping is key to visual perception

The Muller-Lyer illusion

The Gestalt school• Grouping is key to visual perception• Elements in a collection can have properties

that result from relationships • “The whole is greater than the sum of its parts”

The Gestalt school• Grouping is key to visual perception• Elements in a collection can have properties

that result from relationships • “The whole is greater than the sum of its parts”

subjective contours occlusion

familiar configuration

http://en.wikipedia.org/wiki/Gestalt_psychology

Figure-ground discrimination

The ultimate Gestalt?

Gestalt factors

• These factors make intuitive sense, but are very difficult to translate into algorithms

Gestalt factors• They may be hard to put into algorithms, but

understanding them can come in useful for interface design

Gestalt factors• They may be hard to put into algorithms, but

understanding them can come in useful for interface design

Segmentation as clustering

Source: K. Grauman

Segmentation as clustering

Source: K. Grauman

Different clustering strategies• Agglomerative clustering

• Start with each point in a separate cluster• At each iteration, merge two of the “closest” clusters

• Divisive clustering• Start with all points grouped into a single cluster• At each iteration, split the “largest” cluster

• K-means clustering• Iterate: assign points to clusters, compute means

• K-medoids• Same as k-means, only cluster center cannot be computed

by averaging• The “medoid” of each cluster is the most centrally located

point in that cluster (i.e., point with lowest average distance to the other points)

Image Intensity-based clusters Color-based clusters

K-Means clustering• K-means clustering based on intensity or

color is essentially vector quantization of the image attributes• Clusters don’t have to be spatially coherent

K-Means clustering• K-means clustering based on intensity or

color is essentially vector quantization of the image attributes• Clusters don’t have to be spatially coherent

• Clustering based on (r,g,b,x,y) values enforces more spatial coherence

K-Means pros and cons• Pros

• Simple and fast• Converges to a local minimum of the error function

• Cons• Need to pick K• Sensitive to initialization• Sensitive to outliers• Only finds “spherical”

clusters

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

Mean shift segmentation• An advanced and versatile technique for

clustering-based segmentation

D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.

• The mean shift algorithm seeks a mode or local maximum of density of a given distribution

• Choose a search window (width and location)• Compute the mean of the data in the search window• Center the search window at the new mean location • Repeat until convergence

Mean shift algorithm

Region ofinterest

Center ofmass

Mean Shiftvector

Mean shift

Slide by Y. Ukrainitz & B. Sarel

Region ofinterest

Center ofmass

Mean Shiftvector

Mean shift

Slide by Y. Ukrainitz & B. Sarel

Region ofinterest

Center ofmass

Mean Shiftvector

Mean shift

Slide by Y. Ukrainitz & B. Sarel

Region ofinterest

Center ofmass

Mean Shiftvector

Mean shift

Slide by Y. Ukrainitz & B. Sarel

Region ofinterest

Center ofmass

Mean Shiftvector

Mean shift

Slide by Y. Ukrainitz & B. Sarel

Region ofinterest

Center ofmass

Mean Shiftvector

Mean shift

Slide by Y. Ukrainitz & B. Sarel

Region ofinterest

Center ofmass

Mean shift

Slide by Y. Ukrainitz & B. Sarel

• Cluster: all data points in the attraction basin of a mode

• Attraction basin: the region for which all trajectories lead to the same mode

Mean shift clustering

Slide by Y. Ukrainitz & B. Sarel

• Find features (color, gradients, texture, etc)• Initialize windows at individual pixel locations• Perform mean shift for each window until convergence• Merge windows that end up near the same “peak” or mode

Mean shift clustering/segmentation

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

Mean shift segmentation results

More results

More results

Mean shift pros and cons• Pros

• Does not assume spherical clusters• Just a single parameter (window size) • Finds variable number of modes• Robust to outliers

• Cons• Output depends on window size• Computationally expensive• Does not scale well with dimension of feature space

Graph-based segmentation• Represent features and their relationships

using a graph• Cut the graph to get subgraphs with strong

interior links and weaker exterior links

Images as graphs

• Node for every pixel• Edge between every pair of pixels (or every pair

of “sufficiently close” pixels)• Each edge is weighted by the affinity or

similarity of the two nodes

wiji

j

Source: S. Seitz

Segmentation by graph partitioning

• Break Graph into Segments• Delete links that cross between segments• Easiest to break links that have low affinity

– similar pixels should be in the same segments– dissimilar pixels should be in different segments

A B C

Source: S. Seitz

wiji

j

Measuring Affinity

Intensity

Color

Distance

aff x, y( )= exp − 12σ i

2⎛ ⎝

⎞ ⎠ I x( )− I y( ) 2( )⎧

⎨ ⎩

⎫ ⎬ ⎭

aff x, y( )= exp − 12σ d

2⎛ ⎝

⎞ ⎠ x − y 2( )⎧

⎨ ⎩

⎫ ⎬ ⎭

aff x, y( )= exp − 12σ t

2⎛ ⎝

⎞ ⎠ c x( )− c y( ) 2( )⎧

⎨ ⎩

⎫ ⎬ ⎭

Scale affects affinity• Small σ: group only nearby points• Large σ: group far-away points

Graph cut

• Set of edges whose removal makes a graph disconnected

• Cost of a cut: sum of weights of cut edges• A graph cut gives us a segmentation

• What is a “good” graph cut and how do we find one?

A B

Source: S. Seitz

Graph cut

Graph cut

Affinity matrix Block detection

* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University

Multi-way graph cut

Minimum cut• We can do segmentation by finding the

minimum cut in a graph• Efficient algorithms exist for doing this

• Drawback: minimum cut tends to cut off very small, isolated components

* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Minimum cut• We can do segmentation by finding the

minimum cut in a graph• Efficient algorithms exist for doing this

• Drawback: minimum cut tends to cut off very small, isolated components

Ideal Cut

Cuts with lesser weightthan the ideal cut

* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Normalized cut• A minimum cut penalizes large segments• This can be fixed by normalizing the cut by

component size• The normalized cut cost is:

• The exact solution is NP-hard but an approximation can be computed by solving a generalized eigenvalue problem

assoc(A, V) = sum of weights of all edges in V that touch A

),(),(

),(),(

VBassocBAcut

VAassocBAcut

+

J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000

Example results

Example results

Using texture features for segmentation• Texture descriptor is

vector of filter bank outputs

J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

Using texture features for segmentation• Texture descriptor is

vector of filter bank outputs

• Textons are found by clustering

J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

Using texture features for segmentation• Texture descriptor is

vector of filter bank outputs

• Textons are found by clustering

• Affinities are given by similarities of textonhistograms over windows given by the “local scale” of the texture

J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

The importance of scale

J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

Example results

J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation". IJCV 43(1),7-27,2001.

• Pros• Generic framework, can be used with many different

features and affinity formulations

• Cons• High storage requirement and time complexity• Bias towards partitioning into equal segments

* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Normalized cuts: Pro and con

Integrating top-down and bottom-up segmentation

Z.W. Tu, X.R. Chen, A.L. Yuille, and S.C. Zhu. Image parsing: unifying segmentation, detection and recognition. IJCV 63(2), 113-140, 2005.

Image parsing• Define generative models for text and faces

• Deformable spline-based templates for characters

• PCA model for faces

• Top-down: propose a model for a given region• Bottom-up: verify the consistency of the model

with image features

Image parsing

Data-driven Markov Chain Monte Carlo

Example results

Example results