1

Grouping with Bias

Stella X. Yu 1,2 Jianbo Shi 1

Robotics Institute1

Carnegie Mellon UniversityCenter for the Neural Basis of Cognition2

2

What Is It About?

Incorporating prior knowledge into groupingUnitary generative modelGlobal configurations: partially labelled data and object modelsAttention

ComputationEfficient solution in a graph partitioning framework

GoalsBridge the gap between generative models and discriminative modelsBridge the gap between formulation and computation

3

Grouping with Markov Random Fields

MRF: data structure = data generation model + segmentation model

)(log)|(log);(min XpXfpfXE −−=

Segmentation is to find a partitioning of an image, with generative models explaining each partition.

Generative models constrain the continuous observation data, the segmentation model constrains the discrete states.

The solution sought is the most probable state, or the state of the lowest energy.

f1 f2Image as

observation f

Texture models

X1 X2Grouping as

state X

[Geman & Geman, 84, …]

4

Grouping with Spectral Graph Partitioning

SGP: data structure = a weighted graph, weights describing data affinity

)deg()deg(),(

),(min21

2121 VV

VVcutVVNcut

⋅= ∑∑∈ ∈

=1 2

),(),( 21Vp Vq

qpWVVcut

∑∑∈ ∈

=1

),()deg( 1Vp Vq

qpWV

Segmentation is to find a node partitioning of a relational graph, with minimum total cut-off affinity.

Discriminative models are used to evaluate the weights between nodes.

The solution sought is the cuts of the minimum energy.

[Shi & Malik, 97; Perona & Freeman, 98; Malik et al, 01, …]

5

Solving MRF by Graph Partitioning

Some simple MRF models can be translated into graph partitioning

∑∑ ∑ +=∈ p

pppp pNq

qpqp fXUXXWfXE ),(),();(min)(

,

Unitary measuresBinary relationships

L1 L2

[Greig et al, 89; Ferrari et al, 95; Boykov et al, 98; Roy & Cox, 98; Ishikawa & Geiger, 98, …]

6

Comparison of Two Approaches

Formulation ComputationPros \ Cons

Simulation: e.g. Gibbs samplerParameter estimation is hardDifficult to compute probabilityConvergence is very slowOnly local optimum

Generative modelsBayesian interpretation General local interaction

Sensitive to model mismatch

MarkovRandom

Fields

Spectral decompositionFast and robustGlobal optimum

Discriminative modelsNo models requiredLack prior to guide grouping

GraphPartitioning

7

Prior Knowledge in Grouping

Global Configuration ConstraintsLocal Constraints

Unitary generative models

Object models:What to look for

Attention:Where to look for

Red foreground Partial grouping Spatial attention

How to encode them in discriminative models, e.g. SGP?

8

Review: Segmentation on Relational Graphs

G = (V, E, A, R)V: each node denotes a pixelE: each edge denotes a pixel-pixel relationship A: each weight measures pairwise similarityR: each weight measures pairwise dissimilarity

Dual criteria on dual measuresMaximize within-group A and between-group RMinimize between-group A and within-group R

Segmentation = node partitioningbreak V into disjoint sets V1 , V2 ; so that cut-off attraction is small cut-off repulsion is large

9

Review: Energy Function Formulation

∉∈

=l

ll Vu

VuuX

,0,1

)( Group indicators

Weight matrixRDRAW +−= RDR +−

ADD = RD+ Degree matrix

)deg()deg(,)1( 121 V

VXXy =−−= ααα Change of variables

DyyWyy

DXXWXX

XXNassoc TT

t tTt

tT

t ==∑=

2

121 ),(

Energy function as aRayleigh quotient

yDyWDyyWyy

T

T

1max λ=⇒ Eigenvector as solution

10

Review: Eigenvector as a Solution

The derivation holds so long as 121 =+ XX

ααα −=−−= 121)1( XXXy

The eigenvector solution is a linear transformation, scaled and offsetversion of the probabilistic membership indicator for one group.

If y is well separated, then two groups are well defined;otherwise, the separation is ambiguous

Solution ywell separated

Solution yambiguous

stimulus

11

Interaction: from Gaussian to Mexican Hat

RDRAW +−= RDR +−

Attraction

Repulsion

Attraction

2

2

12

1

2

21

2

2

)(

2

12

)(

⋅

−−

−−

−= σσ

σσ

σσ

jiji ffff

ij eeW

2

2

12

1

2

2

)(

2

1

⋅

−−

σσ

σ

σσ

ji ff

e

With repulsion, negative correlations in MRF formulations can be translated into graph partitioning formulations directly.

12

Encoding Bias: Unitary Preference

Introduce dummy nodesExpand the node setSoft Constraints

Foreground

−

=0000

)1(: T

a

aB

BAA γ

γγPreference of red pixels to be foreground,black pixels tobe background

−

=0

0)1(

:r

rB

BRR T

r

rγγ

γγ

γ controls the relative weighting between data and preference.Background

13

Encoding Bias: Partial Grouping

Introduce partial grouping solutionAssign a particular group labelusing dummy nodesHard ConstraintsManual selection:

pixels marked w/ the same color are in one group

Coloring dummy nodes is to assigna particular grouplabel

Background

Foreground

)()(),()( 2211 qXpXqXpX ==

z Qy yM T == or 0

M is the constraint spaceQ is the reduced solution space.

]0,,0,1,0,,0,1,0,0[

0

0)()(

LLL −=

=

=−

T

T

m

ym

qypy

AllQMQM =⊕⊥ ,

Linear:

14

Encoding Bias: Constraining Solution Space

Clustering with Partially labelled data

Segmentation withobject models

z Qy =0=yM T

S: translated versions of a shape

Find the eigenshape Q of SConstraining y = Q z allows usto segment out this particularshape in an image.

General form of constraints: 0)( =Ψ y

15

Encoding Bias: Attention

ModulationConnections for some nodesare enhanced / weakenedWeights at attentionalhotspot are less distorted

Spatial Attention:center region isanalyzed w/ morediscrimination

16

Representations of BiasF

G

1 2 3 4 5 86 7 9

Partial grouping{3, 4}, {6, 7}{9, G}

Preference{F, 1,2,4}{G, 5,7,8}

Attention{5, 6}{8, 9}

Soft constraints ModulationHard constraints

17

Constrained Optimization

DyyWyyyNassoc T

T

=)( 0.. =yMts T Constrained optimization

zQyUUIQ

VUMT

T

=−=

Σ= Constraint spaceFeasible solution space

zDQQzzWQQzzNassoc TT

TT

)()()( = Unconstrained optimization

yyWDQ T λ=−1 Eigensolution available

Rank(Q) = # of nodes - # of independent constraintsProblem: Unconstrained affinity matrix becomes denser

18

Results: Preference and Partial Grouping

Grouping w/o bias:Each is one group

M: Hard constraintsData: three stripes

O : hard constraints∆ : soft constraintsF/G : Filled / empty

Grouping w/ bias:Left and right are one group

U: Basis of constraint spaceM = U Σ VT

19

Results: Preference and Partial Grouping

Hard constraints

Soft constraints

data

prior

20

Results: Partial Grouping

1st Eigenvector 2nd Eigenvector 3rd Eigenvector

Image and manually set partial grouping

First row: Grouping w/o bias

Second row:Grouping w/ bias

The pumpkin starts to emerge as a whole from the background regardless of its surface markings.

21

Results: Figure Detection with Soft Constraints

Background Foreground Difference

Difference Thresholdedused as soft constraints

Grouping of foreground Grouping of foregroundwith bias

22

Results: Spatial Attention

attraction

+repulsion

+bias

Bias for A Bias for R

Grouping with BiasWhat Is It About?Grouping with Markov Random FieldsGrouping with Spectral Graph PartitioningSolving MRF by Graph PartitioningComparison of Two ApproachesPrior Knowledge in GroupingReview: Segmentation on Relational GraphsReview: Energy Function FormulationReview: Eigenvector as a SolutionInteraction: from Gaussian to Mexican HatEncoding Bias: Unitary PreferenceEncoding Bias: Partial GroupingEncoding Bias: Constraining Solution SpaceEncoding Bias: AttentionRepresentations of BiasConstrained OptimizationResults: Preference and Partial GroupingResults: Preference and Partial GroupingResults: Partial GroupingResults: Figure Detection with Soft ConstraintsResults: Spatial Attention