Computational Modeling of Visual Perception The Image ...strider/vis-notes/Seg_Lecture_final.pdf · Image Segmentation COSC 6390D/PSYC 6750B Computational Modeling of Visual Perception

COSC 6390D/PSYC 6750B Computational Modeling of Visual PerceptioCOSC 6390D/PSYC 6750B Computational Modeling of Visual PerceptionnImage SegmentationImage Segmentation

Paco Paco Estrada, Feb. 2006Estrada, Feb. 2006

Computational Modeling of Visual PerceptionComputational Modeling of Visual Perception

The Image Segmentation ProblemThe Image Segmentation Problem

Francisco J. Estrada, Feb. 10, 2006Francisco J. Estrada, Feb. 10, 2006Questions? Comments?Questions? Comments?ee--mail: mail: pacopaco@@elderlabelderlab..yorkuyorku.ca.ca



Suppose I give you the following task:Suppose I give you the following task:

Divide the image into pieces, Divide the image into pieces, where each piece represents a where each piece represents a distinguished thing in the image. It distinguished thing in the image. It is important that all of the pieces is important that all of the pieces have approximately equal have approximately equal importance. The number of things importance. The number of things in the image is up to you. in the image is up to you. Something between 2 and 20 Something between 2 and 20 should be reasonable.should be reasonable.11

11-- Instructions for segmenting images for the Berkeley SegmentatioInstructions for segmenting images for the Berkeley Segmentation Database, see Martin et. al. [MFTM 01]n Database, see Martin et. al. [MFTM 01]



Results from a few observers who performed this task:Results from a few observers who performed this task:



Image Segmentation (in very general terms!):Image Segmentation (in very general terms!):

-- Partitioning the image into salient regionsPartitioning the image into salient regions

-- Salient regions (at least for human observers) tend to represenSalient regions (at least for human observers) tend to represent individual objects,t individual objects,object parts, or individual surfacesobject parts, or individual surfaces

-- But note: Humans use all sorts of tricks to perform this task!But note: Humans use all sorts of tricks to perform this task!

Could we segment this imageCould we segment this image22

without knowing what it is we’rewithout knowing what it is we’reLooking at?Looking at?

22-- Prof. Richard L. Gregory’s famous Dalmatian ImageProf. Richard L. Gregory’s famous Dalmatian Image



BottomBottom--up Image Segmentation:up Image Segmentation:

-- Partitioning the image into regions that have homogeneous appeaPartitioning the image into regions that have homogeneous appearancerance

-- Intended as a preIntended as a pre--processing stage, so we can’t use object recognition.processing stage, so we can’t use object recognition.(in fact, we would like to use the segmentation to help with r(in fact, we would like to use the segmentation to help with recognition!)ecognition!)

-- Assume little or no knowledge about the contents of the imageAssume little or no knowledge about the contents of the image

InputInput SegmentationSegmentation33

SegmentationSegmentation33InputInput

33-- Produced by the MeanProduced by the Mean--Shift algorithm (see Shift algorithm (see Comaniciu Comaniciu & & MeerMeer, 2002 [CM02]), 2002 [CM02])



Why do we think this is useful?Why do we think this is useful?

-- Good segmentation should help with recognition, tracking, imageGood segmentation should help with recognition, tracking, image databasedatabaseretrieval, and image compression among other highretrieval, and image compression among other high--level vision tasks.level vision tasks.

“Detect and recognize objects in this image” “Detect and recognize objects in this image” –– very hard!very hard!Unless, of course, you’ve seen THIS particular tiger before, in Unless, of course, you’ve seen THIS particular tiger before, in a similar pose,a similar pose,

but we’ll get to that when we talk about object recognition…but we’ll get to that when we talk about object recognition…

“What is the object represented by the pixels labeled in red?”“What is the object represented by the pixels labeled in red?”–– somewhat more manageablesomewhat more manageable

Though this is an idealized example. In practice we’re likely toThough this is an idealized example. In practice we’re likely to get several regions get several regions that we hope can be joined to form the tiger’s outline.that we hope can be joined to form the tiger’s outline.



What we’re doing today:What we’re doing today:

-- Define the bottomDefine the bottom--up segmentation problem (we just did!)up segmentation problem (we just did!)

-- Talk about lowTalk about low--level visual similarity: How do we determine that two pixels level visual similarity: How do we determine that two pixels should be in the same region?should be in the same region?

-- How do we encode pixel similarity?How do we encode pixel similarity?

-- Review of segmentation methodsReview of segmentation methods

* Early techniques (* Early techniques (thresholdingthresholding, region growing), region growing)* Feature space methods* Feature space methods* Graph Theoretic methods* Graph Theoretic methods

-- How do we evaluate the quality of a segmentation?How do we evaluate the quality of a segmentation?



The segmentation task… segmented!The segmentation task… segmented!

It is useful to make a distinction between two subIt is useful to make a distinction between two sub--problems that mustproblems that mustbe addressed by bottombe addressed by bottom--up segmentation algorithms:up segmentation algorithms:

1.1.-- Determining a suitable measure of similarity between image featDetermining a suitable measure of similarity between image features.ures.

2.2.-- Developing a clustering (or grouping) algorithm that generates Developing a clustering (or grouping) algorithm that generates regionsregionsusing the proposed similarity measure.using the proposed similarity measure.

In general, the clustering part can be made to work with difIn general, the clustering part can be made to work with differentferentsimilarity measures, and it is in this part that we will spend msimilarity measures, and it is in this part that we will spend most of theost of thelecture, but first a few comments on similarity measures…lecture, but first a few comments on similarity measures…



Similarity Measures. What cues are at hand?Similarity Measures. What cues are at hand?

-- Image brightnessImage brightness -- ColourColour

-- Edge energyEdge energy -- TextureTexture

-- Image Position (spatial info)Image Position (spatial info)

-- MotionMotion

-- Stereo disparityStereo disparity And now, the trick is deciding how to combine these cues. And now, the trick is deciding how to combine these cues. For some insight seeFor some insight see MalikMalik et al. [MBLS01]et al. [MBLS01]



Segmenting Images… early techniquesSegmenting Images… early techniques

GrayGray--level level thresholdingthresholding

-- Groups of similar pixels appear as bumps in the brightness histGroups of similar pixels appear as bumps in the brightness histogramogram

-- Split the histogram at localSplit the histogram at local--minimaminima

-- Label pixels w.r.t. which bump they belong inLabel pixels w.r.t. which bump they belong in




GrayGray--level level thresholdingthresholding

-- Notice bleeding problem. Also, regions are not connectedNotice bleeding problem. Also, regions are not connected

-- How many thresholds to use?How many thresholds to use?

-- Simple, but can be useful in suitably constrained environmentsSimple, but can be useful in suitably constrained environments




Region growingRegion growing

-- Start with a set of seed regions (perhaps individual pixels)Start with a set of seed regions (perhaps individual pixels)

-- Progressively grow regions by adding to them neighboring pixelsProgressively grow regions by adding to them neighboring pixels whosewhosesimilarity is above some threshold.similarity is above some threshold.

-- Stop when all the image has been labeledStop when all the image has been labeled

-- SeeSee BeveridgeBeveridge et al. [BGKHR89], Pal and Pal [PP93]et al. [BGKHR89], Pal and Pal [PP93]



Brief aside: overBrief aside: over-- and underand under--segmentationsegmentation

Two common terms used to describe general problems with segmentaTwo common terms used to describe general problems with segmentationstions44

Good segmentationGood segmentation(human)(human)

UndersegmentationUndersegmentation(Mean(Mean--Shift)Shift)

OversegmentationOversegmentation(Mean(Mean--Shift)Shift)4 4 -- Notice that underNotice that under-- and over and over

segmentation are related, but not segmentation are related, but not identical to the problem of identical to the problem of deciding how much detail should deciding how much detail should be in a segmentationbe in a segmentation



Segmenting Images… feature space methodsSegmenting Images… feature space methods

-- Select a set of image features, say position {x,y}, Select a set of image features, say position {x,y}, colour colour {R,G,B}, and{R,G,B}, anda set of filter responses {fa set of filter responses {f11(x,y)…(x,y)…ffkk(x,y)}(x,y)}

-- For each pixel pFor each pixel pii((xxii,,yyii) form a ) form a feature vectorfeature vector

1

2

( , )...

( , )

i

i

i

i

ii

i i

i i

xyRGB

vf x y

f x y

⎛ ⎞⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟=⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠

-- This vector represents a point in an This vector represents a point in an NN--dimensionaldimensionalfeature spacefeature space. . NN is the number of featuresis the number of features

-- The feature vectors for similar pixels shouldThe feature vectors for similar pixels shouldoccupy nearby locations in this feature spaceoccupy nearby locations in this feature space

-- Thus, homogeneous image regions become denseThus, homogeneous image regions become denseclouds of feature vectors in feature space.clouds of feature vectors in feature space.




-- Quick example, let’s look at an RGB feature space for a simpleQuick example, let’s look at an RGB feature space for a simpleimage:image:

-- The feature space has dimension 3. Notice that the principal imThe feature space has dimension 3. Notice that the principal imageageregions generate dense clusters in feature space.regions generate dense clusters in feature space.




KK--means clusteringmeans clustering55

-- Compute the feature space vectorsCompute the feature space vectors

-- Randomly select Randomly select KK cluster centers in feature space (they don’t havecluster centers in feature space (they don’t haveto correspond to any of the feature vectors, but seeto correspond to any of the feature vectors, but see FischlerFischler &&BollesBolles [FB81]).[FB81]).

-- Iterate until convergence:Iterate until convergence:

* Assign feature vectors to the closest cluster center* Assign feature vectors to the closest cluster center

* Re* Re--compute the cluster centers as a (weighted) mean ofcompute the cluster centers as a (weighted) mean ofthe feature vectors assigned to each clusterthe feature vectors assigned to each cluster

-- Label pixels according to the cluster their feature vectors belLabel pixels according to the cluster their feature vectors belong to.ong to.

5 5 –– See Forsyth & Ponce, Ch. 14See Forsyth & Ponce, Ch. 14




KK--means clusteringmeans clustering

-- 8 iterations of the K8 iterations of the K--means procedure, K=5means procedure, K=5




KK--means clusteringmeans clustering

-- Effect of random initialization, K=5Effect of random initialization, K=5

-- Effect of the choice of KEffect of the choice of K

K=3K=3 K=5K=5 K=8K=8 K=15K=15




Mixture of Mixture of Gaussians Gaussians Model (see Carson et al., ‘Model (see Carson et al., ‘Blobworld’Blobworld’ [CBGM02])[CBGM02])

-- Model the distribution of feature vectors as a mixture of Model the distribution of feature vectors as a mixture of GaussiansGaussians

1

1

( | ) ( | , ),

1

K

k k kk

k

kk

p v M N vπ µ

π

=

=

= Σ

=

∑

∑

-- Where Where µµkk and and ΣΣkk are the mean and covariance matrix of are the mean and covariance matrix of GaussiansGaussians,,and and ππkk are the mixing proportions.are the mixing proportions.

-- Model parameters can be fit using maximumModel parameters can be fit using maximum--likelihood (the Elikelihood (the E--MMalgorithm).algorithm).




Mixture of Mixture of Gaussians Gaussians ModelModel

-- Pixels are assigned to the Pixels are assigned to the Gaussian Gaussian component that has thecomponent that has themaximum maximum ownership probabilityownership probability

k

( | , )arg max( | )

k i k ki

i

N vlabelp v M

π µ⎡ ⎤Σ= ⎢ ⎥

⎣ ⎦

-- Blobworld Blobworld uses uses colourcolour, texture, and position as features, texture, and position as features

See [CBGM02] and See [CBGM02] and http://http://elibelib..cscs..berkeleyberkeley..eduedu//blobworldblobworld//




MeanMean--Shift algorithm (Shift algorithm (Comaniciu Comaniciu & & MeerMeer, see [CM02]), see [CM02])

-- Starting at each feature vector , initialize a hypotheticalStarting at each feature vector , initialize a hypothetical clusterclustercenter center

iv0 ic v=

-- Iteratively update to be the (weighted) mean of all featurIteratively update to be the (weighted) mean of all feature vectorse vectorswithin a small search window within a small search window S( ) S( ) centered at centered at

tctc

( )1

( )

( )

( )j t

j t

t j jv S c

tt j

v S c

w c v vc

w c v∈

+

∈

−

=−

∑

∑

tc

-- The search window is usually a (hyper) sphere, ellipsoid, or cuThe search window is usually a (hyper) sphere, ellipsoid, or cube in be in RRdd. . The label of pixel The label of pixel i i is set to the value of at convergence.is set to the value of at convergence.ic




MeanMean--Shift algorithmShift algorithm

-- The meanThe mean--shift iterationshift iterationconverges to a region of converges to a region of locally maximal densitylocally maximal densityin feature space.in feature space.





-- A point of locally maximalA point of locally maximaldensity forms a basin ofdensity forms a basin ofattraction for nearbyattraction for nearbyfeature vectorsfeature vectors66

6 6 –– But notice that the range of But notice that the range of attraction of this basin attraction of this basin depends on the size and depends on the size and shape of the search windowshape of the search window

-- These These domains ofdomains ofconvergenceconvergence define thedefine thesegments producedsegments producedby meanby mean--shiftshift





-- Sample segmentationsSample segmentations77

Input imageInput image smaller search windowsmaller search window larger search windowlarger search window

7 7 –– Using the EDISON system, see [EDISON]Using the EDISON system, see [EDISON]



Segmenting Images… graphSegmenting Images… graph--theoretic methodstheoretic methods

Images as graphs…Images as graphs…

-- An image I(x,y) is equivalentAn image I(x,y) is equivalentto a graph G(V,E)to a graph G(V,E)

Original image I(x,y)Original image I(x,y)

-- E is a set of E is a set of edgesedges linking neighboring nodes together. The linking neighboring nodes together. The weightweight or or strengthstrength ofofthe edge is proportional to the similarity between the verticethe edge is proportional to the similarity between the vertices it joins together.s it joins together.

-- V is a set of V is a set of verticesvertices or or nodesnodes, , each node represents one image each node represents one image element (e.g. individual pixels)element (e.g. individual pixels)

Graph G(V,E)Graph G(V,E)




Images as graphs…Images as graphs…

-- The image elements can be individual pixels, small regions, orThe image elements can be individual pixels, small regions, orother types of image features.other types of image features.

-- Usually, only elements within a small neighborhood are connecteUsually, only elements within a small neighborhood are connected.d.This provides spatial coherence and has computational advantagThis provides spatial coherence and has computational advantages.es.

-- Edge weight is also calledEdge weight is also called pairwisepairwise similaritysimilarity, or, or pairwisepairwise affinityaffinity..

-- In general, given a graph G(V,E) graphIn general, given a graph G(V,E) graph--based segmentationbased segmentationmethods attempt to find groups of nodes in G that are stronglymethods attempt to find groups of nodes in G that are stronglyconnected to one another, but weakly connected to the rest of connected to one another, but weakly connected to the rest of thethegraph.graph.




Minimal spanning trees (Minimal spanning trees (MSTsMSTs))

-- Tree that connects all vertices of the graphTree that connects all vertices of the graphwith the minimum total weight (sum of linkwith the minimum total weight (sum of linkweight in the tree)weight in the tree)

-- Can be constructed efficiently usingCan be constructed efficiently usingKruskal’s Kruskal’s algorithm:algorithm:

•• Start with a disconnected graphStart with a disconnected graph

•• Add edges in increasing order ofAdd edges in increasing order ofweight as long as doing so weight as long as doing so doesn’t introduce a cycledoesn’t introduce a cycle

•• Stop when all the vertices areStop when all the vertices areconnectedconnected

2

4

5

1

3

6

8 2

15 15

74

89

7

Minimal Spanning Tree forMinimal Spanning Tree forthis graphthis graph




The Local Variation algorithm (see The Local Variation algorithm (see Felzenszwalb Felzenszwalb & & Huttenlocher Huttenlocher [FH04])[FH04])

-- Partition the image so that for any pair of regions the variatiPartition the image so that for any pair of regions the variationonbetween the regions should be larger than the variation withinbetween the regions should be larger than the variation within thetheregions.regions.

-- Extension of Extension of Kruskal’s Kruskal’s algorithmalgorithm88::

•• Add edges one at a time in order of increasing weight. Maintain Add edges one at a time in order of increasing weight. Maintain a list ofa list ofdisconnected disconnected MSTsMSTs

•• For each MST For each MST CCii compute a threshold , where w(compute a threshold , where w(CCii))is the maximum weight in the spanning tree, |is the maximum weight in the spanning tree, |CCii| is the number of pixels| is the number of pixelsin in CCii, and k>0 is a user, and k>0 is a user--defined constant)defined constant)

( ) ( ) / | |i i iT C w C k C= +

•• If the next edge to be added joins two separate If the next edge to be added joins two separate MSTsMSTs, the , the MSTs MSTs are mergedare mergedonly ifonly if

( , ) min( ( ), ( ))k l i jw x x T C T C≤

8 8 –– See also Jepson [Jep04] for more detailsSee also Jepson [Jep04] for more details




The Local Variation algorithmThe Local Variation algorithm

-- Sample segmentationsSample segmentations99

Input imageInput image smaller smaller kk larger larger kk

9 9 –– Generated using the implementation provided by the authors of [Generated using the implementation provided by the authors of [FH04], see [FH_code]FH04], see [FH_code]




Graph CutsGraph Cuts

-- A A cutcut through a graph is defined asthrough a graph is defined asthe total weight of the links that mustthe total weight of the links that mustbe removed to divide the graph intobe removed to divide the graph intotwo separate components.

AA

BB two separate components.

,,

( , ) i ji A j B

cut A B w∈ ∈

= ∑





Minimum Cut method (see Wu & Leahy [WL93])Minimum Cut method (see Wu & Leahy [WL93])

10 10 10

10

10

10

10 10 10

10

10

10

1010

10

10

.1 .1

.1

.1

.1.1

.1

.1

-- Find the cut through the graph thatFind the cut through the graph thathas the overall minimum weighthas the overall minimum weight

,( , ) ( ( , ))min

A BMinCut A B cut A B=

-- Should correspond to the subset ofShould correspond to the subset ofedges of least weight that can beedges of least weight that can beremoved to partition the graphremoved to partition the graph

-- Since weight encodes similarity, thisSince weight encodes similarity, thisshould be equivalent to partitioningshould be equivalent to partitioningthe graph along the boundary of leastthe graph along the boundary of leastsimilaritysimilarity( , )MinCut A B = 8*(.1) .8=




Minimum Cut methodMinimum Cut method

-- Can be computed efficientlyCan be computed efficiently 10 10 10

10

10

10

10 10 10

10

10

10

1010

10

10

5 5

5

5

55

5

5

-- However, it has a preference for However, it has a preference for shortshortboundaries. Sometimes picks aboundaries. Sometimes picks atrivial partitiontrivial partition

( , )cut A B = 8*(5) 40=




Minimum Cut methodMinimum Cut method

-- Can be computed efficientlyCan be computed efficiently 10 10 10

10

10

10

10 10 10

10

10

10

1010

10

10

5 5

5

5

55

5

5

-- However, it has a preference for However, it has a preference for shortshortboundaries. Sometimes picks aboundaries. Sometimes picks atrivial partitiontrivial partition

-- We have to somehow constrain theWe have to somehow constrain thesolution to avoid such trivial cutssolution to avoid such trivial cuts ( , )cut A B = 8*(5) 40=

( , )MinCut A B = 2*(10) 20=




Normalized Cuts (see Shi & Normalized Cuts (see Shi & Malik Malik [SM00])[SM00])

-- Find the cut through the graph thatFind the cut through the graph thatminimizes the normalized cutminimizes the normalized cutmeasuremeasure

AA

BB

-- Here the term Here the term assoc(A,V)assoc(A,V) is the totalis the totalweight of the connections betweenweight of the connections betweenthe region the region AA and the rest of the nodesand the rest of the nodesin the graphin the graph

( , )NCut A B =( , )cut A B ( , )cut A B

+( , )assoc A V ( , )assoc B V( , )cut A B ( , )cut A B

( , )assoc A V ( , )assoc B V


,, ,( , )

( , )i j

i ji A j V v v E

assoc A V w∈ ∈ ∈

= ∑




Normalized CutsNormalized Cuts

10 10 10

10

10

10

10 10 10

10

10

10

1010

10

10

5 5

5

5

55

5

5

( , )cut A B = 2*(10) 20=

( , )assoc A V = 2*(10) 20=

( , )assoc B V = 16*10 8*5 200+ =

( , )NCut A B = 20 20+20 200 1.1=

-- Trivial cuts such as this one don’tTrivial cuts such as this one don’tminimize the normalized cutminimize the normalized cut





10 10 10

10

10

10

10 10 10

10

10

10

1010

10

10

5 5

5

5

55

5

5( , )NCut A B = 40 40+160 80

.75=

( , )cut A B = 8*(5) 40=

( , )assoc A V = 12*10 8*4 160+ =

( , )assoc B V = 4*10 8*5 80+ =

-- The normalization by The normalization by assoc(R,V)assoc(R,V)should (in general) prevent the biasshould (in general) prevent the biastoward short boundariestoward short boundaries





-- Solving for the optimal Solving for the optimal NCutNCut exactly is NPexactly is NP--completecomplete

-- However, we can obtain an approximate solution. Start with an However, we can obtain an approximate solution. Start with an affinity matrixaffinity matrix WW s.t.s.t.W(i,j) W(i,j) contains the weight of the edge linkingcontains the weight of the edge linkingnodes nodes ii and and jj

-- Notice that if we have an image (n x m) pixels in size, Notice that if we have an image (n x m) pixels in size, WW will bewill bean an (n*m)(n*m)22 matrix. Fortunately, it is sparsematrix. Fortunately, it is sparse

-- Also, define a diagonal matrix Also, define a diagonal matrix DD s.t. s.t. ,( , ) i jj

D i i w=∑





-- Shi and Shi and Malik Malik show that an approximate solution is given by theshow that an approximate solution is given by theeigenvector with the smallest eigenvector with the smallest eigenvalue eigenvalue of the systemof the system1010

( )D W y Dyλ− =

-- Furthermore, succeeding eigenvectors can be used to split previFurthermore, succeeding eigenvectors can be used to split previousouspartitionspartitions

-- General General NCutNCut procedure (there are several variants!)procedure (there are several variants!)

•• Compute W and DCompute W and D

•• Solve the above system for the Solve the above system for the kk eigenvectors with smallest eigenvectors with smallest eigenvalueseigenvalues

•• Threshold each eigenvector, from the 2Threshold each eigenvector, from the 2ndnd smallest on, to obtain a partition smallest on, to obtain a partition of the imageof the image

•• The segmentation is the intersection of the The segmentation is the intersection of the kk partitions generated in this waypartitions generated in this way10 10 –– Meila Meila and Shi [MS01] present a probabilistic interpretation of and Shi [MS01] present a probabilistic interpretation of NCuts NCuts in terms of random walksin terms of random walks





22ndnd eigenvectoreigenvector

Input imageInput image

33rdrd eigenvectoreigenvector

Segmentation with 25 segmentsSegmentation with 25 segments1111

44thth eigenvectoreigenvector

11 11 –– Generated using the implementation provided by at [Generated using the implementation provided by at [NCutNCut_code]_code]




SS--T MinT Min--Cut (see Cut (see Boykov Boykov & & Kolmogorov Kolmogorov [BK01], and [BK01], and BoykovBoykov & Jolly [BJ01])& Jolly [BJ01])

-- Reduce the problem of trivial cuts by introducing two special nReduce the problem of trivial cuts by introducing two special nodesodescalledcalled sourcesource (S) (S) andand sinksink (T)(T)

T

S -- SS and and TT are linked to some image nodesare linked to some image nodesby links of very large weight (so that theyby links of very large weight (so that theywill never be selected in a cut)will never be selected in a cut)

-- Find the minimum cut that separates theFind the minimum cut that separates thesource from the sinksource from the sink

-- Notice that the problem is deciding howNotice that the problem is deciding howto connect to connect SS and and TT to the image nodesto the image nodes

Min SMin S--T CutT Cut




SS--T MinT Min--CutCut

UserUser--selected pixels connected to selected pixels connected to SS (red)(red)and pixels connected to and pixels connected to TT (cyan)(cyan)

Minimum SMinimum S--T cutT cut

-- If we have a good ‘guess’ (or a human observer) to tell us hot If we have a good ‘guess’ (or a human observer) to tell us hot to linkto linkthe source and sink to the image, we will get an optimal segmethe source and sink to the image, we will get an optimal segmentationntation





Source pixels (purple) andSource pixels (purple) andsink pixels (yellow)sink pixels (yellow)

Minimum SMinimum S--T cutT cut

-- With a bad guess, things don’t go so well…With a bad guess, things don’t go so well…

-- And, how do we get this guess to begin with?And, how do we get this guess to begin with?





-- Good choices for the sets of image nodes connected to Good choices for the sets of image nodes connected to SS and and TT shouldshouldsatisfy:satisfy:

•• Each set should be sufficiently large (otherwise, we get a triviEach set should be sufficiently large (otherwise, we get a trivial cutal cutthat contains only the image nodes in either the set for that contains only the image nodes in either the set for SS or the set for or the set for T)T)

•• Both sets should be fully contained within ‘natural’ segments (tBoth sets should be fully contained within ‘natural’ segments (this ishis isbecause the sets themselves will never be partitioned. Think abobecause the sets themselves will never be partitioned. Think about theut theshirt leaking into the wall in the segmentation on the previoushirt leaking into the wall in the segmentation on the previous slide)s slide)

•• Many different sets for Many different sets for SS and and TT should be tried to identify most of the should be tried to identify most of the salient boundaries in the imagesalient boundaries in the image




SESE--MinCut MinCut (see Estrada, Jepson & (see Estrada, Jepson & ChennubhotlaChennubhotla [EJC04])[EJC04])

-- Use spectral clustering to identify small groups of similar pixUse spectral clustering to identify small groups of similar pixels.els.These groups defineThese groups define seed regionsseed regions for Sfor S--T T MinCutMinCut

Seed regions found through Seed regions found through spectral clustering. spectral clustering.

Combinations of these are Combinations of these are used as source and sink used as source and sink regions for Sregions for S--T T MinCutMinCut

Input imageInput image Final segmentation Final segmentation combines the partitions combines the partitions

generated with each generated with each separate Sseparate S--T cut.T cut.




SESE--MinCutMinCut

-- Sample segmentationsSample segmentations

Input imageInput image Seed regionsSeed regions SegmentationSegmentation




SWA SWA -- Segmentation by Weighted Aggregation (see Segmentation by Weighted Aggregation (see Galun Galun et al. [GSBB03])et al. [GSBB03])

-- The methods covered so far are mostly singleThe methods covered so far are mostly single--scale, and localscale, and local

Global information removes orGlobal information removes orReduces the uncertaintyReduces the uncertainty

Locally, we may haveLocally, we may haveuncertainty about theuncertainty about thesegmentationsegmentation

-- Makes sense to analyze the image at multiple scales for segmentMakes sense to analyze the image at multiple scales for segmentationation




SWA SWA -- Segmentation by Weighted AggregationSegmentation by Weighted Aggregation

-- Very general description of SWA:Very general description of SWA:

•• Compute a fineCompute a fine--scale graph with nodes corresponding to individual pixelsscale graph with nodes corresponding to individual pixels

•• Progressively coarsen the graph by contracting similar nodesProgressively coarsen the graph by contracting similar nodes

•• Each node above the individual pixel level contains statistics Each node above the individual pixel level contains statistics aboutaboutintensity, texture, and shape of the elements that compose itintensity, texture, and shape of the elements that compose it at the nextat the nextfiner levelfiner level

,, ,( , )

,, ,( , )

( )( )( )

i j i j

i j i j

i jv S v S v v E

i jv v S v v E

wE SE SN S w

∈ ∉ ∈

∈ ∈

= =∑

∑•• For a region For a region SS the aggregation process minimizesthe aggregation process minimizes

•• This is similar to theThis is similar to the NCutNCut measure. It normalizes the sum of weights alongmeasure. It normalizes the sum of weights alongthe region’s boundary by the sum of internal weightsthe region’s boundary by the sum of internal weights





-- Schematic diagram of SWASchematic diagram of SWA1212

12 12 –– From From Ronen Basri’s Ronen Basri’s SWA SWA webpage webpage -- http://www.wisdom.http://www.wisdom.weizmannweizmann.ac..ac.ilil/~/~ronenronen/segmentation.html/segmentation.html





-- Sample Sample segmentationssegmentationsalsoalso from (12)from (12)



Segmenting Images… quick recapSegmenting Images… quick recap

-- First: Choose a suitable measure of similarityFirst: Choose a suitable measure of similarity

-- Choose your Choose your favourite favourite segmentation method:segmentation method:

•• ThresholdingThresholding, simple region growing, simple region growing•• KK--means clusteringmeans clustering•• Feature space clustering (Mean Shift algorithm)Feature space clustering (Mean Shift algorithm)•• GraphGraph--based methods (Local Variation, Normalized Cuts, SEbased methods (Local Variation, Normalized Cuts, SE--MinCutMinCut, SWA), SWA)

-- Have a segmentation!… now, what do we do with it?Have a segmentation!… now, what do we do with it?



Segmenting Images… a couple of questionsSegmenting Images… a couple of questions

-- Given a (likely imperfect) segmentation, how do we use it to imGiven a (likely imperfect) segmentation, how do we use it to improveproverecognition, tracking, and other vision tasks?recognition, tracking, and other vision tasks?

-- How do we put together segmentation and grouping (boundaryHow do we put together segmentation and grouping (boundarydetection) methods? Are they the same thing working on differendetection) methods? Are they the same thing working on differenttdata?data?

-- For that matter. How do we evaluate bottomFor that matter. How do we evaluate bottom--up segmentationup segmentationresults?results?



Segmentation evaluationSegmentation evaluation

-- Columns 2Columns 2--5 show 5 show segmentations by segmentations by different algorithms different algorithms (each column (each column corresponds to 1 corresponds to 1 method). Which method). Which segmentations are segmentations are better?better?

SESE--MinCutMinCut

MeanMean--ShiftShift

Local VariationLocal Variation

NCutsNCuts




-- Visual evaluation can be tricky, how can we be sure?Visual evaluation can be tricky, how can we be sure?

-- Quantitative evaluationQuantitative evaluation

•• Task dependent (i.e. how much does a segmentation help in solvinTask dependent (i.e. how much does a segmentation help in solving someg someparticular problem?)particular problem?)

•• Evaluation w.r.t. a standard image database with groundEvaluation w.r.t. a standard image database with ground--truthtruth

-- We have a standard image database: the Berkeley Segmentation We have a standard image database: the Berkeley Segmentation Database (BSD)Database (BSD)




-- 200 training images, 100200 training images, 100testing imagestesting images

-- At least 4 human groundAt least 4 human ground--truthtruthsegmentations per imagesegmentations per image

-- Interesting observation: HumanInteresting observation: Humansegmentations are consistentsegmentations are consistentwith one another.with one another.

-- Most variation concerns ‘level ofMost variation concerns ‘level ofdetail’, notice that this isdetail’, notice that this isdifferent from overdifferent from over-- and underand under--segmentationsegmentation




-- We can compare automatic segmentations against the groundWe can compare automatic segmentations against the groundtruth, and see which methods work better overalltruth, and see which methods work better overall

-- Need an appropriate comparison metricNeed an appropriate comparison metric

-- Martin et al. (see [MFTM01]) propose regionMartin et al. (see [MFTM01]) propose region--based measures ofbased measures ofsegmentation consistency. These are either too harsh, orsegmentation consistency. These are either too harsh, orinsensitive to overinsensitive to over-- and underand under--segmentationsegmentation

-- We will use the region boundaries instead, and compute We will use the region boundaries instead, and compute precisionprecisionand and recall recall (see Martin [Mar02], and Estrada & Jepson [EJ05])(see Martin [Mar02], and Estrada & Jepson [EJ05])




-- Precision: Percentage of Precision: Percentage of detecteddetected boundary pixels that correspond toboundary pixels that correspond togroundground--truthtruth (human) boundary pixels(human) boundary pixels

-- Recall: Percentage of Recall: Percentage of groundground--truthtruth (human) boundary pixels that were(human) boundary pixels that weredetecteddetected in the automatic segmentationin the automatic segmentation

Original imageOriginal image Overlayed Overlayed human boundarieshuman boundaries

MeanMean--shift overshift over--segmentedsegmented MeanMean--shift undershift under--segmentedsegmented




-- Take a segmentation algorithmTake a segmentation algorithm

-- Run it on the 100 test images, and compute precision and recallRun it on the 100 test images, and compute precision and recallw.r.t. the human segmentationsw.r.t. the human segmentations

-- The median precision/recall values over the 100 test imagesThe median precision/recall values over the 100 test imagesgive an indication of segmentation qualitygive an indication of segmentation quality

-- Systematically vary the algorithm’s parameters to create a curvSystematically vary the algorithm’s parameters to create a curveethat fully characterizes the performance of the algorithmthat fully characterizes the performance of the algorithm

-- Compare the Compare the tuning curvestuning curves of different algorithms directly!of different algorithms directly!




-- Does this agree with our visualDoes this agree with our visualevaluation?evaluation?

-- What’s the deal with CannyWhat’s the deal with Cannyedges? Given these results, whyedges? Given these results, whyeven bother doingeven bother doingsegmentation?segmentation?

-- What about this gap betweenWhat about this gap betweenhuman performance andhuman performance andautomatic segmentationautomatic segmentationresults? How do we get there?results? How do we get there?

Tuning curves for several segmentation Tuning curves for several segmentation methods (see Estrada [Est05])methods (see Estrada [Est05])




HighestHighestrecallrecall

MidMid--curvecurve HighestHighestprecisionprecision

SESE--MinCutMinCut

MeanMean--shiftshift

Local Local variationvariation

NCutNCut

Canny Canny edgesedges

-- Notice the large variationNotice the large variationbetween segmentations atbetween segmentations atdifferent points along the tuningdifferent points along the tuningcurvescurves

-- Usually, the better Usually, the better segmentations occur close tosegmentations occur close tothe middle of the tuning curvethe middle of the tuning curve(a compromise between(a compromise betweenprecision and recall)precision and recall)

-- Canny edges don’t partition theCanny edges don’t partition theimage. We would need to groupimage. We would need to groupthem (which is quite anotherthem (which is quite anotherproblem) to generate imageproblem) to generate imagepartitionspartitions




-- What about the gap in performance between human observers andWhat about the gap in performance between human observers andautomatic segmentation methods?automatic segmentation methods?

-- It could be that we just don’t know how to exploit available loIt could be that we just don’t know how to exploit available loww--levellevelcues appropriatelycues appropriately

-- Perhaps more likely, human observers use highPerhaps more likely, human observers use high--level knowledge tolevel knowledge tocomplete the segmentation task when lowcomplete the segmentation task when low--level information is ambiguouslevel information is ambiguousor missingor missing

-- We should think about this: How much can we expect to be able tWe should think about this: How much can we expect to be able tooimprove bottomimprove bottom--up segmentation methods? How good is good enough?up segmentation methods? How good is good enough?




-- How well can we do on a purelyHow well can we do on a purelyperceptual problem? (i.e. where perceptual problem? (i.e. where object recognition or high levelobject recognition or high levelknowledge are not required)knowledge are not required)

-- 107 regions of uniform brightness plus107 regions of uniform brightness plusfractal noise with ground truthfractal noise with ground truth

-- Same evaluation procedure as withSame evaluation procedure as withBSD imagesBSD images

InputInput GroundGround--truthtruth SESE--MinCutMinCut




Tuning curves on fractal images (see Tuning curves on fractal images (see Estrada [Est05])Estrada [Est05])



Segmentation… a few final thoughtsSegmentation… a few final thoughts

-- Current segmentation algorithms seem reasonably capableCurrent segmentation algorithms seem reasonably capable

-- We don’t (yet) have many direct applications of segmentation, hWe don’t (yet) have many direct applications of segmentation, howeveroweverit is quite popular in medical imagingit is quite popular in medical imaging

-- Recent research has focused on the use of more midRecent research has focused on the use of more mid--level informationlevel information(i.e. shape priors, segmentation of known object classes, etc. (i.e. shape priors, segmentation of known object classes, etc. see forsee forexample Kumar et al. [KTZ05])example Kumar et al. [KTZ05])

-- Ultimately, achieving human performance is likely to require aUltimately, achieving human performance is likely to require acombination of bottomcombination of bottom--up and topup and top--down processing. We don’t knowdown processing. We don’t knowhow this should work at the present timehow this should work at the present time



References[BGKHR99] Beveridge, Griffith, Kohler, Hanson, and Riseman, “Segmenting Images Using Localized Histograms and Region Merging”,

IJCV, 1989

[BJ01] Boykov and Jolly, “Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images”,ICCV, 2001

[BK01] Boykov and Kolmogorov, “An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision”,Lecture Notes in Computer Science, 2001

[CM02] Comaniciu and Meer, “Mean Shift: A Robust Approach toward Feature Space Analysis”, PAMI, 2002

[CBGM02] Carson, Belongie, Greenspan, and Malik,” Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying”, PAMI, 2002

[EDISON] Georgescu and Christoudias: The Edge Detection and Image SegmentatiON (EDISON) system, Robust Image UnderstandingLaboratory, Rutgers University, http://www.caip.rutgers.edu/riul/research/code.html

[EJC04] Estrada, Jepson, and Chennubhotla, “Spectral Embedding and Min-Cut for Image Segmentation”, BMVC, 2004

[EJ05] Estrada and Jepson, “Quantitative Evaluation of a Novel Image Segmentation Algorithm”, CVPR, 2005

[Est05] Estrada, “Advances in Computational Image Segmentation and Grouping ”, Ph.D. Thesis, University of Toronto, 2005

[FB81] Fischler and Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography”, Comm. of the ACM, 1981.

[FH04] Felzenszwalb and Huttenlocher, “Efficient Graph-Based Image Segmentation”, IJCV, 2004.

[FH_code] Felzenszwalb and Huttenlocher: Image segmentation software, MIT A.I. Lab, http://www.ai.mit.edu/people/pff/seg/seg.html

[GSBB03] Galun, Sharon, Basri and Brandt, “Segmentation by Multiscale Aggregation of Filter Responses and Shape Elements”,ICCV, 2003

[Jep04] Jepson: Lecture notes on Image Segmentation, http://www.cs.utoronto.ca/~jepson/csc2503/

[KTZ05] Kumar, Torr, and Zisserman, “OBJ CUT”, CVPR, 2005



[Mar02] Martin, “An Empirical Approach to Grouping and Segmentation”, Ph.D. Thesis, UC Berkeley, 2002

[MBLS01] Malik, Belongie, Leung, and Shi, “Contour and Texture Analysis for Image Segmentation”, IJCV, 2001

[MFTM01] Martin, Fowlkes, Tal, and Malik, “A Database of Human-segmented Natural Images and its Application to EvaluatingSegmentation Algorithms and Measuring Ecological Statistics”, ICCV, 2001

[MS02] Meila and Shi, “Learning Segmentation by Random Walks”, NIPS, 2002

[NCut_code] Cour, Yu, and Shi: Normalized Cuts Matlab code, Computer and Information Science, Penn State University,http://www.cis.upenn.edu/~jshi/software/

[PP93] Pal and Pal, “A Review on Image Segmentation Techniques”, Pattern Recognition, 1993

[SM00] Shi and Malik, “Normalized Cuts and Image Segmentation”, PAMI, 2000

[WL93] Wu and Leahy, “An Optimal Graph-Theoretic Approach to Data Clustering: Theory and its Application to Image Segmentation”,PAMI, 1993

Documents

Computational Modeling of Visual Perception The Image ...strider/vis-notes/Seg_Lecture_final.pdf · Image Segmentation COSC 6390D/PSYC 6750B Computational Modeling of Visual Perception