Eitan Sharon, CVPR’04
Segmentation by Weighted Aggregation
Eitan Sharon
Division of Applied Mathematics, Brown University
Eitan Sharon, Meirav Galun, Ronen Basri, Achi Brandt
Hierarchical Adaptive Texture Segmentationor
Segmentation by Weighted Aggregation (SWA)
Dept. of CS and Applied Mathematics,The Weizmann Institute of Science,
Rehovot, Israel
Eitan Sharon, CVPR’04
Image Segmentation
Eitan Sharon, CVPR’04
Local Uncertainty
Eitan Sharon, CVPR’04
Global Certainty
Eitan Sharon, CVPR’04
Local Uncertainty
Eitan Sharon, CVPR’04
Global Certainty
Eitan Sharon, CVPR’04
Hierarchy in SWA
Eitan Sharon, CVPR’04
Segmentation by Weighted Aggregation
A multiscale algorithm:• Optimizes a global measure• Returns a full hierarchy of segments• Linear complexity• Combines multiscale measurements:
– Texture– Boundary integrity
Eitan Sharon, CVPR’04
The Pixel Graph
Couplings
Reflect intensity similarity
Low contrast –strong coupling
High contrast –weak coupling
{ }ijw
Eitan Sharon, CVPR’04
Normalized-Cut Measure
∉∈
=SiSi
ui 01
Eitan Sharon, CVPR’04
Normalized-Cut Measure
2( ) ( )ij i ji j
E S w u u≠
= −∑
∉∈
=SiSi
ui 01
Eitan Sharon, CVPR’04
Normalized-Cut Measure
2( ) ( )ij i ji j
E S w u u≠
= −∑
∉∈
=SiSi
ui 01
( ) ij i jN S w u u=∑
Eitan Sharon, CVPR’04
Normalized-Cut Measure
2( ) ( )ij i ji j
E S w u u≠
= −∑
∉∈
=SiSi
ui 01
( ) ij i jN S w u u=∑
( )( )( )
E SSN S
Γ =
Minimize:
Eitan Sharon, CVPR’04
Normalized-Cut Measure
High-energy cut
Minimize:
( )( )( )
E SSN S
Γ =
Eitan Sharon, CVPR’04
Normalized-Cut MeasureLow-energy cut
Minimize:
( )( )( )
E SSN S
Γ =
Eitan Sharon, CVPR’04
Matrix Formulation
,( ) ikk k iij
ij
w i jl
w i j≠
== − ≠
∑Define matrix by
Define matrix byW 0ijw > 0iiw =
L
We minimize ( ) 12
T
T
u Luuu Wu
Γ =
Eitan Sharon, CVPR’04
Coarsening the Minimization Problem
iu ju
Eitan Sharon, CVPR’04
Coarsening – Choosing a Coarse Grid
iu julU
kU
Representative subset
1 2( , ,..., )NU U U U=
Eitan Sharon, CVPR’04
Coarsening –Interpolation Matrix
iu ju
2
11
2
.
..
Nn
u
U
P
u
Uu
U
For a salient segment (globally minimizing solutions):
P ( )n N× , sparse interpolation matrix
kU
lU
Eitan Sharon, CVPR’04
Coarsening – Matrix Formulation
( )( )
( ) 1 12 2
T TT
T T T
U P LP Uu Luuu Wu U P WP U
Γ = ≈
Given such an appropriate interpolation matrix P
Existence of from Algebraic Multigrid (AMG)for solving the equivalent Eigen-Value problem:
P
Lu Duλ= for minimal 0λ >solve
Eitan Sharon, CVPR’04
Weighted Aggregation
ijwi
jjlp
aggregate k aggregate l
ijk ik jli j
l p pwW≠
= ∑
klWikp
Eitan Sharon, CVPR’04
Importance of Soft Relations
Eitan Sharon, CVPR’04
SWA
Linear in # of points(a few dozen operations per point)
Detects the salient segments
Hierarchical structure
Eitan Sharon, CVPR’04
Coarse-Scale Measurements
• Average intensities of aggregates
• Multiscale intensity-variances of aggregates
• Multiscale shape-moments of aggregates
• Boundary alignment between aggregates
Eitan Sharon, CVPR’04
Coarse Measurements for Texture
Eitan Sharon, CVPR’04
A Chicken and Egg Problem
Problem:Coarse measurements mix neighboring statistics
Solution:Support of measurements is determined as the segmentation process proceeds
Hey, I was here first
Eitan Sharon, CVPR’04
Adaptive vs. Rigid Measurements
AveragingOur algorithm - SWA Geometric
Original
Eitan Sharon, CVPR’04
Our algorithm - SWA
Adaptive vs. Rigid Measurements
Interpolation Geometric
Original
Eitan Sharon, CVPR’04
Recursive Measurements: Intensity
i
ikpk
aggregate k
j
iq intensity of pixel i
ik ii
kik
i
p qQ
p=∑∑
average intensityof aggregate
Eitan Sharon, CVPR’04
Use Averages to Modify the Graph
Eitan Sharon, CVPR’04
Hierarchy in SWA
•Average intensity•Texture•Shape
Eitan Sharon, CVPR’04
Texture Examples
Eitan Sharon, CVPR’04
Isotropic Texture in SWA
( ) ( )22k kk
V I I= −Intensity Variance
Isotropic Texture of aggregate –average of variances in all scales
Eitan Sharon, CVPR’04
Oriented Texture in SWA
Shape Moments
Oriented Texture of aggregate –orientation, width and length in all scales
2 2, , , ,x y xy x y
•center of mass•width•length•orientation
with Meirav Galun
Eitan Sharon, CVPR’04
Implementation
• Our SWA algorithm (CVPR’00 + CVPR’01 + orientations Texture)run-time: << 1 seconds.
images on a Pentium III 1000MHz PC:200 200×
400 400×
run-time: 2-3 seconds.
Eitan Sharon, CVPR’04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR’04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR’04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR’04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR’04
Isotropic Texture
Our Algorithm (SWA)
Eitan Sharon, CVPR’04
Our Algorithm (SWA)
Our previous
Eitan Sharon, CVPR’04
Our Algorithm (SWA)
Our previous
Eitan Sharon, CVPR’04
Our Algorithm (SWA)
Our previous
Eitan Sharon, CVPR’04
Our Algorithm (SWA)
Our previous