Improved Moves for Truncated Convex Models

M. Pawan Kumar

Philip Torr

AimEfficient, accurate MAP for truncated convex models

V1 V2 … … …

… … … … …

… … … … …

… … … … Vn

Random Variables V = { V1, V2, …, Vn}

Edges E define neighbourhood

Aim

Va Vb

li

lkab;ik

Accurate, efficient MAP for truncated convex models

ab;ik = wab min{ d(i-k), M }

ab;ik

i-k

wab is non-negative

Truncated Linear

i-k

ab;ik

Truncated Quadratic

d(.) is convexa;i b;k

MotivationLow-level Vision

• Smoothly varying regions

• Sharp edges between regions

min{ |i-k|, M}

Boykov, Veksler & Zabih 1998

Well-researched !!

Things We Know• NP-hard problem - Can only get approximation

• Best possible integrality gap - LP relaxation

Manokaran et al., 2008

• Solve using TRW-S, DD, PP

Slower than graph-cuts

• Use Range Move - Veksler, 2007

None of the guarantees of LP

Real MotivationGaps in Move-Making Literature

LPMove-Making

Potts

Truncated Linear

Truncated Quadratic

2

Multiplicative Bounds

2 + √2

O(√M)

Chekuri et al., 2001

Real MotivationGaps in Move-Making Literature

LPMove-Making

Potts

Truncated Linear

Truncated Quadratic

2

Multiplicative Bounds

2

2 + √2 2M

O(√M) -

Boykov, Veksler and Zabih, 1999

Real MotivationGaps in Move-Making Literature

LPMove-Making

Potts

Truncated Linear

Truncated Quadratic

2

Multiplicative Bounds

2

2 + √2 4

O(√M) -

Gupta and Tardos, 2000

Real MotivationGaps in Move-Making Literature

LPMove-Making

Potts

Truncated Linear

Truncated Quadratic

2

Multiplicative Bounds

2

2 + √2 4

O(√M) 2M

Komodakis and Tziritas, 2005

Real MotivationGaps in Move-Making Literature

LPMove-Making

Potts

Truncated Linear

Truncated Quadratic

2

Multiplicative Bounds

2

2 + √2

O(√M)

2 + √2

O(√M)

Outline

• Move Space

• Graph Construction

• Sketch of the Analysis

• Results

Move Space

Va Vb

• Initialize the labelling

• Choose interval I of L’ labels

• Each variable can

• Retain old label

• Choose a label from I

• Choose best labelling

Iterate over intervals

Outline

• Move Space

• Graph Construction

• Sketch of the Analysis

• Results

Two Problems

Va Vb

• Choose interval I of L’ labels

• Each variable can

• Retain old label

• Choose a label from I

• Choose best labelling

Large L’ => Non-submodular

Non-submodular

First Problem

Va Vb Submodular problem

Ishikawa, 2003; Veksler, 2007

First Problem

Va Vb Non-submodularProblem

First Problem

Va Vb Submodular problem

Veksler, 2007

First Problem

Va Vb

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

First Problem

Va Vb

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

First Problem

Va Vb

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

First Problem

Va Vb

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

First Problem

Va Vb

Model unary potentials exactly

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

First Problem

Va Vb

Similarly for Vb

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

First Problem

Va Vb

Model convex pairwise costs

am+1

am+2

an

t

am+2

bm+1

bm+2

bn

bm+2

First Problem

Va Vb

Overestimated pairwise potentials

Wanted to model

ab;ik = wab min{ d(i-k), M }

For all li, lk I

Have modelled

ab;ik = wab d(i-k)

For all li, lk I

Second Problem

Va Vb

• Choose interval I of L’ labels

• Each variable can

• Retain old label

• Choose a label from I

• Choose best labelling

Non-submodular problem !!

Second Problem - Case 1

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

s∞ ∞

Both previous labels lie in interval

Second Problem - Case 1

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

s∞ ∞

wab d(i-k)

Second Problem - Case 2

Va Vb

Only previous label of Va lies in interval

am+1

am+2

an

t

bm+1

bm+2

bn

s∞ ub

Second Problem - Case 2

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

ub : unary potential of previous label of Vb

M

s∞ ub

Second Problem - Case 2

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

M

wab d(i-k)

s∞ ub

Second Problem - Case 2

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

M

wab ( d(i-m-1) + M )

s∞ ub

Second Problem - Case 3

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

Only previous label of Vb lies in interval

Second Problem - Case 3

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

sua

∞

ua : unary potential of previous label of Va

M

Second Problem - Case 4

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

Both previous labels do not lie in interval

Second Problem - Case 4

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

sua ub

Pab : pairwise potential for previous labels

ab

Pab

MM

Second Problem - Case 4

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

wab d(i-k)

sua ub

ab

Pab

MM

Second Problem - Case 4

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

wab ( d(i-m-1) + M )

sua ub

ab

Pab

MM

Second Problem - Case 4

Va Vb

am+1

am+2

an

t

bm+1

bm+2

bn

Pab

sua ub

ab

Pab

MM

Graph Construction

Va Vb

Find st-MINCUT. Retain old labellingif energy increases.

am+1

am+2

an

bm+1

bm+2

bn

t

ITERATE

Outline

• Move Space

• Graph Construction

• Sketch of the Analysis

• Results

Analysis

Va Vb

Current labelling f(.)

QC ≤ Q’C

Va Vb

Global Optimum f*(.)

QP

Previous labelling f’(.)

Va Vb

Analysis

Va Vb

Current labelling f(.)

QC ≤ Q’C

Va Vb

Partially Optimal f’’(.) Previous labelling f’(.)

Va Vb

Q’0≤

Analysis

Va Vb

Current labelling f(.)

QP - Q’C

Va Vb

Partially Optimal f’’(.) Previous labelling f’(.)

Va Vb

QP- Q’0≥

Analysis

Va Vb

Current labelling f(.)

QP - Q’C

Va Vb

Partially Optimal f’’(.) Local Optimal f’(.)

Va Vb

QP- Q’0≤ 0 ≤ 0

Analysis

Va Vb

Current labelling f(.)

Va Vb

Partially Optimal f’’(.) Local Optimal f’(.)

Va Vb

QP- Q’0 ≤ 0Take expectation over all intervals

AnalysisTruncated Linear

QP ≤ 2 + max 2M , L’L’ MQ*

L’ = M 4Gupta and Tardos, 2000

L’ = √2M 2 + √2

Truncated Quadratic

QP ≤ O(√M)Q*

L’ = √M

Outline

• Move Space

• Graph Construction

• Sketch of the Analysis

• Results

Synthetic Data - Truncated Linear

Faster than TRW-S Comparable to Range Moves

With LP Relaxation guarantees

Time (sec)

Energy

Synthetic Data - Truncated Quadratic

Faster than TRW-S Comparable to Range Moves

With LP Relaxation guarantees

Time (sec)

Energy

Stereo Correspondence

Disparity Map

Unary Potential: Similarity of pixel colour

Pairwise Potential: Truncated convex

Stereo Correspondence

Algo Energy1 Time1 Energy2 Time2

Swap 3678200 18.48 3707268 20.25

Exp 3677950 11.73 3687874 8.79

TRW-S 3677578 131.65 3679563 332.94

BP 3789486 272.06 5180705 331.36

Range 3686844 97.23 3679552 141.78

Our 3613003 120.14 3679552 191.20

Teddy

Stereo Correspondence

Algo Energy1 Time1 Energy2 Time2

Swap 3678200 18.48 3707268 20.25

Exp 3677950 11.73 3687874 8.79

TRW-S 3677578 131.65 3679563 332.94

BP 3789486 272.06 5180705 331.36

Range 3686844 97.23 3679552 141.78

Our 3613003 120.14 3679552 191.20

Teddy

Stereo Correspondence

Algo Energy1 Time1 Energy2 Time2

Swap 645227 28.86 709120 20.04

Exp 634931 9.52 723360 9.78

TRW-S 634720 94.86 651696 226.07

BP 662108 170.67 2155759 244.71

Range 634720 39.75 651696 80.40

Our 634720 66.13 651696 80.70

Tsukuba

Summary

• Moves that give LP guarantees

• Similar results to TRW-S

• Faster than TRW-S because of graph cuts

Questions Not Yet Answered

• Move-making gives LP guarantees– True for all MAP estimation problems?

• Huber function? Parallel Imaging Problem?

• Primal-dual method?

• Solving more complex relaxations?

Questions?

Improved Moves for Truncated Convex Models

Kumar and Torr, NIPS 2008

http://www.robots.ox.ac.uk/~pawan/