Beyond Trees: MRF Inference via Outer-Planar Decomposition

Beyond Trees: MRF Inference via Outer-Planar Decomposition[(To appear) CVPR 10]Dhruv BatraCarnegie Mellon University

Joint work: Andrew Gallagher (Eastman Kodak), Devi Parikh (TTI-C), Tsuhan Chen (Cornell, CMU)

1Labelling Problems in VisionSegmentation[Ren et al. ECCV 06], [Boykov and Jolly, ICCV 01], [Batra et al. BMVC 08], [Rother et al., SIGGRAPH 04]

[He et al. ECCV 06], [Yang et al. CVPR 07], [Shotton et al. IJCV 09], [Batra et al. CVPR 08].

2

(C) Dhruv Batra

Labelling Problems in VisionGeometric Labelling[Hoiem et al. IJCV 07], [Hoiem et al. CVPR 08], [Saxena PAMI 08], [Ramalingam et al. CVPR 08].

3(C) Dhruv Batra

Labelling Problems in VisionName-Face Association[Berg et al. CVPR 04, Phd-Thesis 07], [Gallagher et al. CVPR 08].4(C) Dhruv Batra

President George W. Bush makes a statement in the Rose Garden while Secretary of Defense Donald Rumsfeld looks on, July 23, 2003. Rumsfeld said the United States would release graphic photographs of the dead sons of Saddam Hussein to prove they were killed by American troops. Photo by Larry Downing/ReutersBritish director Sam Mendes and his partner actress Kate Winslet arrive at the London premiere of The Road to Perdition, September 18, 2002. The films stars Tom Hanks as a Chicago hit man who has a separate family life and co-stars Paul Newman and Jude Law. REUTERS/Dan Chung

Mildred and Lisa

LisaMildredLabelling Problems in VisionStereo (Disparity Labelling)[Roy and Cox, ICCV 98], [Boykov et al. CVPR 97, PAMI 01], [Scharstein et al. PAMI 02].

Motion Flow[Boykov et al. PAMI 01]

Denoising[Sebastiani et al. Signal Proc. 97], [Roth et al. IJCV 09], [Li et al. ECCV 08], [Ishikawa CVPR 09].

5(C) Dhruv Batra

Left imageRight image

Disparity map

5Markov Random FieldsSet of random variables

Pairwise MRF

MAP Inference6(C) Dhruv Batra X1X2Xn6InferenceMAP problem

In general NP-hard(C) Dhruv Batra 7Approximate AlgorithmsExact Algorithms for subclasses Trees [Pearl, AAAI 82] Submodular Energies [Hammer 65, Kolmogorov PAMI 04] Outer-planar graphs [Schraudolph NIPS 08] Loopy BP[Pearl, 88] Tree-Reweighted MP[Wainwright 05, Kolmogorov 06, Komodakis 07] Outer-Planar Decomposition (OPD) Decomposition MethodsDecompose into subproblems(C) Dhruv Batra 8Decomposition MethodsComparisons:Max-Product BP[Pearl, 88]

Tree-Reweighted MP [Wainwright 05, Kolmogorov 06, Komodakis 07]

(C) Dhruv Batra 9Decomposition MethodsProgression(C) Dhruv Batra 10BPTRWOPDOuter-planarityG is outer-planarAllows a planar embeddingAll nodes are accessible from outsideLie on an unbounded external faceAdd an extra node connected to all other. Result should be planar.

ExamplesTreesA lot more, e.g.

(C) Dhruv Batra 11123412341234Planar InferencePlanar-Cut Construction[Schraudolph et al. NIPS 08]

E(X1,X2,X3,X4) X1 = 1 X2 = 0 X3 = 0 X4 = 1

E(1,0,0,1) = 8 + constCost of st-cut = 8Source (0) X1X2X4X3Planar InferencePlanar-CutSource (0) X1X2X4X3Planar-cutOuter-planarityE(X1,X2,X3,X4) Outer-planar DecompositionOuter-planarity is a strong constraintThese are not OP:

So now?We will leverage this new subclass to propose a new approximate MAP inference algorithmOuter-Planar Decomposition (OPD)

(C) Dhruv Batra 14Very important for visionOuter-planar DecompositionOPD(C) Dhruv Batra 15Non-outerplanar graph Outer-planar DecompositionOPD(C) Dhruv Batra 16Non-outerplanar graph OPDPlanar graphsOuter-planar DecompositionOPD

Message-PassingOPD-MP: Generalizes BPOPD-DD: Generalizes TRW-DD

OPD-T: Generalizes TRW-T OPD-S: Generalizes TRW-S

(C) Dhruv Batra 17[Dual Decomposition, Komodakis, ICCV 07, CVPR 09][Wainwright, Inf. Theory 05][Kolmogorov, PAMI 06]Outer-planar DecompositionOPD

(C) Dhruv Batra 18Agreement VariablesmessagesOuter-planar DecompositionOPD-MP: Message(C) Dhruv Batra 19Non-outerplanar graph OPDAgreement VariablesOuter-planar DecompositionOPD-MP: Message(C) Dhruv Batra 20Non-outerplanar graph OPDAgreement VariablesMin-marginalMessageGuaranteesOPD-MPsame as BPNone!Contains BP as a special case

OPD-DDCan be analyzed as projected subgradient ascent on dual. Guaranteed to converge. Contains TRW as a special case

Guaranteed to perform better than TRW!

(C) Dhruv Batra 21DecompositionsWhere do sub-graphs come from?

In general: NP-hardMaximum outer-planar sub-graph problem

(C) Dhruv Batra 22DecompositionsWhere do sub-graphs come from?

Heuristic:Start with a spanning treeAdd edges till maximal outer-planarRemove edges from graph, repeat

(C) Dhruv Batra 23Outer-Planar DecompositionFor Grids:

Theorem [Goncalves STOC 05]: Every planar graph can be decomposed into 2 outer-planar graphsLinear-time algorithm

(C) Dhruv Batra 24ExperimentsSynthetic (constructed) energies

Real-world applicationsFace-Gender LabellingSemantic Object SegmentationOptical Flow

(C) Dhruv Batra 25ExperimentsSynthetic parametersRandomly sampled energies from Gaussian dist.(C) Dhruv Batra 26ResultsSynthetic parametersRandomly sampled energies from Gaussian dist.(C) Dhruv Batra 27K4: 2-label problemHarder Problems (More interaction)EnergyLower boundResultsSynthetic parametersRandomly sampled energies from Gaussian dist.(C) Dhruv Batra 2830 x 30 grid: 2-labelResultsGender-Face Assignment(C) Dhruv Batra 29ResultsMulti-class Object Labelling(C) Dhruv Batra 30ResultsOptical Flow(C) Dhruv Batra 31ArmyWoodel

Take-home MessagesFor people working on these problems:First step towards structures topologically more complex than treesOPD-approximation is guaranteed to be tighter than TRWOPD is useful for hard non-submodular problemsTraditional vision benchmarks might be saturated

For people using BP/TRW:Stop! Use OPD.

(C) Dhruv Batra 32Thank You!33ResultsSynthetic parametersRandomly sampled energies from Gaussian dist.(C) Dhruv Batra 34K50: 2-label problemResultsSynthetic parametersRandomly sampled energies from Gaussian dist.(C) Dhruv Batra 35K50: 4-label problemOuter-planar DecompositionOPD-DD: Message(C) Dhruv Batra 36Non-outerplanar graph OPDAgreement VariablesDecompositionsHeuristic:Able to find 2-subgraph decompositions surprisingly often

(C) Dhruv Batra 37Decomposition MethodsCertificate property

Decomposition Lower Bound

(C) Dhruv Batra 38GstGiven graph G, can we push an s-t flow of value > k?Flow-value >kCut-value ABC A?D!E? A?D!DD F>!;!A?D!DD

! "! #! $! %! &!$$'$

$$'%

$$'&

$$'(

$$')

$$'*

$$'+

$%,-./01

23./4356-7

8

89:;

=?!@=

>=?!??

AA=?!??

! "! #! $! %! &!&&'%

&&'(

&&')

&(

&('#

&('%

&('(

&(')*+,-./

01,-2134+5

6

6

789

:;

;

ABC A?D!E? A?D!DD F>!;!A?D!DD

0 10 20 30 40 50150

200

250

300

350Energy

Iterations

0 10 20 30 40 50400

600

800

1000

Energy

Iterations

0 10 20 30 40 50600

800

1000

1200

Energy

Iterations

10 20 30 40 50

270

275

280

285Energy

Iterations

10 20 30 40

780

790

800

810

Energy

Iterations

10 20 30 40 50

1030

1040

1050

1060

1070

Energy

Iterations

Edi (s)+ = t

([Xdi == s]

dO(u)[X

di == s]

|O(u)|

)

dvi() = X d

vid

dvi

V = {v1, . . . , vn}

vid(xi) = ()

E(X ) =p

i=1

Ei(X i)

minXE(X )

pi=1

minX i

Ei(X i)

Eij(0, 0) + Eij(1, 1) Eij(0, 1) + Eij(1, 0)

O(N1.5 logN)

O(N2M)

O(N3)

O(NM log(N))

O(N +M)

O(N2M)

Eij(0, 0) + Eij(1, 1) Eij(0, 1) + Eij(1, 0)

X = argminXL

cC

Ec(Xc)