Division of Applied Mathematics, Brown University

Eitan Sharon, CVPR’04

Segmentation by Weighted Aggregation

Eitan Sharon

Division of Applied Mathematics, Brown University

Eitan Sharon, Meirav Galun, Ronen Basri, Achi Brandt

Hierarchical Adaptive Texture Segmentationor

Segmentation by Weighted Aggregation (SWA)

Dept. of CS and Applied Mathematics,The Weizmann Institute of Science,

Rehovot, Israel

Eitan Sharon, CVPR’04

Image Segmentation

Eitan Sharon, CVPR’04

Local Uncertainty

Eitan Sharon, CVPR’04

Global Certainty

Eitan Sharon, CVPR’04

Local Uncertainty

Eitan Sharon, CVPR’04

Global Certainty

Eitan Sharon, CVPR’04

Hierarchy in SWA

Eitan Sharon, CVPR’04

Segmentation by Weighted Aggregation

A multiscale algorithm:• Optimizes a global measure• Returns a full hierarchy of segments• Linear complexity• Combines multiscale measurements:

– Texture– Boundary integrity

Eitan Sharon, CVPR’04

The Pixel Graph

Couplings

Reflect intensity similarity

Low contrast –strong coupling

High contrast –weak coupling

{ }ijw

Eitan Sharon, CVPR’04

Normalized-Cut Measure

∉∈

=SiSi

ui 01

Eitan Sharon, CVPR’04

Normalized-Cut Measure

2( ) ( )ij i ji j

E S w u u≠

= −∑

∉∈

=SiSi

ui 01

Eitan Sharon, CVPR’04

Normalized-Cut Measure

2( ) ( )ij i ji j

E S w u u≠

= −∑

∉∈

=SiSi

ui 01

( ) ij i jN S w u u=∑

Eitan Sharon, CVPR’04

Normalized-Cut Measure

2( ) ( )ij i ji j

E S w u u≠

= −∑

∉∈

=SiSi

ui 01

( ) ij i jN S w u u=∑

( )( )( )

E SSN S

Γ =

Minimize:

Eitan Sharon, CVPR’04

Normalized-Cut Measure

High-energy cut

Minimize:

( )( )( )

E SSN S

Γ =

Eitan Sharon, CVPR’04

Normalized-Cut MeasureLow-energy cut

Minimize:

( )( )( )

E SSN S

Γ =

Eitan Sharon, CVPR’04

Matrix Formulation

,( ) ikk k iij

ij

w i jl

w i j≠

== − ≠

∑Define matrix by

Define matrix byW 0ijw > 0iiw =

L

We minimize ( ) 12

T

T

u Luuu Wu

Γ =

Eitan Sharon, CVPR’04

Coarsening the Minimization Problem

iu ju

Eitan Sharon, CVPR’04

Coarsening – Choosing a Coarse Grid

iu julU

kU

Representative subset

1 2( , ,..., )NU U U U=

Eitan Sharon, CVPR’04

Coarsening –Interpolation Matrix

iu ju

2

11

2

.

..

Nn

u

U

P

u

Uu

U

For a salient segment (globally minimizing solutions):

P ( )n N× , sparse interpolation matrix

kU

lU

Eitan Sharon, CVPR’04

Coarsening – Matrix Formulation

( )( )

( ) 1 12 2

T TT

T T T

U P LP Uu Luuu Wu U P WP U

Γ = ≈

Given such an appropriate interpolation matrix P

Existence of from Algebraic Multigrid (AMG)for solving the equivalent Eigen-Value problem:

P

Lu Duλ= for minimal 0λ >solve

Eitan Sharon, CVPR’04

Weighted Aggregation

ijwi

jjlp

aggregate k aggregate l

ijk ik jli j

l p pwW≠

= ∑

klWikp

Eitan Sharon, CVPR’04

Importance of Soft Relations

Eitan Sharon, CVPR’04

SWA

Linear in # of points(a few dozen operations per point)

Detects the salient segments

Hierarchical structure

Eitan Sharon, CVPR’04

Coarse-Scale Measurements

• Average intensities of aggregates

• Multiscale intensity-variances of aggregates

• Multiscale shape-moments of aggregates

• Boundary alignment between aggregates

Eitan Sharon, CVPR’04

Coarse Measurements for Texture

Eitan Sharon, CVPR’04

A Chicken and Egg Problem

Problem:Coarse measurements mix neighboring statistics

Solution:Support of measurements is determined as the segmentation process proceeds

Hey, I was here first

Eitan Sharon, CVPR’04

Adaptive vs. Rigid Measurements

AveragingOur algorithm - SWA Geometric

Original

Eitan Sharon, CVPR’04

Our algorithm - SWA

Adaptive vs. Rigid Measurements

Interpolation Geometric

Original

Eitan Sharon, CVPR’04

Recursive Measurements: Intensity

i

ikpk

aggregate k

j

iq intensity of pixel i

ik ii

kik

i

p qQ

p=∑∑

average intensityof aggregate

Eitan Sharon, CVPR’04

Use Averages to Modify the Graph

Eitan Sharon, CVPR’04

Hierarchy in SWA

•Average intensity•Texture•Shape

Eitan Sharon, CVPR’04

Texture Examples

Eitan Sharon, CVPR’04

Isotropic Texture in SWA

( ) ( )22k kk

V I I= −Intensity Variance

Isotropic Texture of aggregate –average of variances in all scales

Eitan Sharon, CVPR’04

Oriented Texture in SWA

Shape Moments

Oriented Texture of aggregate –orientation, width and length in all scales

2 2, , , ,x y xy x y

•center of mass•width•length•orientation

with Meirav Galun

Eitan Sharon, CVPR’04

Implementation

• Our SWA algorithm (CVPR’00 + CVPR’01 + orientations Texture)run-time: << 1 seconds.

images on a Pentium III 1000MHz PC:200 200×

400 400×

run-time: 2-3 seconds.

Eitan Sharon, CVPR’04

Isotropic Texture

Our Algorithm (SWA)

Eitan Sharon, CVPR’04

Isotropic Texture

Our Algorithm (SWA)

Eitan Sharon, CVPR’04

Isotropic Texture

Our Algorithm (SWA)

Eitan Sharon, CVPR’04

Isotropic Texture

Our Algorithm (SWA)

Eitan Sharon, CVPR’04

Isotropic Texture

Our Algorithm (SWA)

Eitan Sharon, CVPR’04

Our Algorithm (SWA)

Our previous

Eitan Sharon, CVPR’04

Our Algorithm (SWA)

Our previous

Eitan Sharon, CVPR’04

Our Algorithm (SWA)

Our previous

Eitan Sharon, CVPR’04

Our Algorithm (SWA)

Our previous