What Energy Functions Can be Minimized UsingGraph Cuts?Shai Bagon

Advanced Topics in Computer Vision June 2010

What is an Energy Function?Ea numbersuggested solutionFor a given problem:Image Segmentation:237-20Useful Energy function:Good solution Low energyTractable Can be minimized

Families of Functions or Outline F2 submodular

Non submodular

F3

Beyond F3

Foreground SelectionLetyi color of ith pixelxi {0,1} BG/FG labels (variables)Given BG/FG scribbles:Pr(xi|yi)=How likely each pixel to be FG/BGPr(xm|xn)=Adjacent pixels should have same labelF2 energy:E(x)=iEi(xi)+ijEij(xi,xj)

SubmodularKnown concept from set-functions:

E(x) = i Ei(xi) + ij Eij (xi, xj), xi {0,1}

1CD0ABxjxi01Eij(xi,xj):What does it mean?B+C-A-D 0

F2 submodularHow toMinimize?E(x) = i Ei(xi) + ij Eij (xi, xj), xi {0,1}

Local beliefs:Data termPrior knowledge: Smoothness term

Graph PartitioningA weighted graph G=( V E w )Special Nodes: s ts-t cut:

Cost of a cut:

Nice property: 1:1 mappings-t cut {0,1}|V|-2 wVEwijst

Graph Partitioning - EnergyE(x) = i Ei(xi) + ij Eij (xi, xj)

stGraph PartitioningijEj(1)D-CB+C-A-DEi(0)C-A

Graph Partitioning - EnergyE(x) = i Ei(xi) + ij Eij (xi, xj)

stGraph PartitioningijEj(1)B+C-A-DEi(0)C-AD-Cst cut binary assignmentcut cost energy of assignmentmin cut Energy min.B=Eij(0,1)

RecapF2 submodular:E(x) = i Ei(xi) + ij Eij (xi, xj)Eij(1,0)+Eij(0,1)Eij(0,0)+Eij(1,1)

Mapping from energy to graph partition

Min Energy = computing min-cutGlobal optimum in poly time for submodular functions!

NextMulti-label F2E(x)=i Ei(xi) + ij Eij(xi,xj) s.t. xi {1,,L}Fusion moves: solving binary sub-problemsApplications to stereo, stitching, segmentation Currentlabelingsuggestedlabeling= FusionSolve Binary problem: xi=0 xi=1

Stereo matching see http://vision.middlebury.edu/stereo/

Ground truthPairwise MRF[Boykov et al. 01]slide by Carsten Rother, ICCV09Input:

Panoramic stitchingslide by Carsten Rother, ICCV09

Panoramic stitchingslide by Pushmeet Kohli, ICCV09

AutoCollagehttp://research.microsoft.com/en-us/um/cambridge/projects/autocollage/[Rother et. al. Siggraph 05 ]

NextMulti-label F2E(x)=i Ei(xi) + ij Eij(xi,xj) s.t. xi {1,,L}Fusion moves: solving binary sub-problemsApplications to stereo, stitching, segmentationNon-submodularBeyond pair-wise interactions: F3

Merging Regionsinput imageregions (Ncuts)edge prob.piweak edgestrong edgepi prob. of boundary being edgeGOAL: Find labeling xi{0,1} that max:ijmin:Taking -log

Merging RegionsAdding and subtracting the same number

Merging RegionsSolving for edges:Consistency constraints: No dangling edgeJwixiNo longer pair-wise:F3

Minimization trickFreedman D., Turek MW, Graph cuts with many pixel interactions: theory and applications to shape modeling. Image Vision Computing 2010

Merging RegionsThe resulting energy:

+ Pair-wise- Non submodular!

Quadratic Pseudo-Boolean OptimizationsijKolmogorov V., Carsten R., Minimizing non-submodular functions with graph cuts a review. PAMI07

Quadratic Pseudo-Boolean Optimization+ All edges with positive capacities- No constraint

Labeling rule:

partial labelingsijt

Quadratic Pseudo-Boolean OptimizationProperties of partial labeling y:1. Let z=FUSE(y,x) E(z)E(x)2. y is subset of optimal y*

y is complete:1. E submodular2. Exists flipping (inference in trees)sijt

QPBO:Probe Node p:What can we say about variables?

r -> is always 0s -> is always equal to qt -> is 0 when q = 1

slide by Pushmeet Kohli, ICCV09QBPO - Probing

0?????

000??

0010?

Probe nodes in an order until energy unchanged Simplified energy preserves global optimality and (sometimes) gives the global minimum

slide by Pushmeet Kohli, ICCV09QBPO - Probing

Merging RegionsResult using QPBO-P:Resultregions (Ncuts)input image

RecapF3 and moreMinimization trickNon submodularQPBO approx. partial labeling

Beyond F3[Kohli et. al. CVPR 07, 08, PAMI 08, IJCV 09]

Image SegmentationE(X) = ci xi + dij |xi-xj|ii,jE: {0,1}n R

0 fg, 1bgn = number of pixels[Boykov and Jolly 01] [Blake et al. 04] [Rother et al.`04]ImageUnary CostSegmentation

Pn Potts Potentials Patch Dictionary (Tree)[slide credits: Kohli]

Pn Potts Potentials E(X) = ci xi + dij |xi-xj| + hp (Xp) ii,jpE: {0,1}n R

0 fg, 1bgn = number of pixels[slide credits: Kohli]

Image SegmentationE(X) = ci xi + dij |xi-xj| + hp (Xp) ii,jImagePairwise SegmentationFinal SegmentationpE: {0,1}n R

0 fg, 1bgn = number of pixels[slide credits: Kohli]

Application: Recognition and Segmentationfrom [Kohli et al. 08]ImageUnaries only TextonBoost [Shotton et al. 06] Pairwise CRF only [Shotton et al. 06] Pn PottsOne super-pixelizationanother super-pixelization

Robust(soft) Pn Potts modelpfrom [Kohli et al. 08]Robust Pn PottsPn Potts

Application: Recognition and SegmentationFrom [Kohli et al. 08]ImageUnaries only TextonBoost [Shotton et al. 06] Pairwise CRF only [Shotton et al. 06] Pn Potts robust Pn Potts robust Pn Potts(different f)One super-pixelizationanother super-pixelization

Same idea for surface-based stereo[Bleyer 10]One input imageGround truth depthStereo with hard-segmentationStereo with robust Pn PottsThis approach gets best result on Middlebury Teddy image-pair:

How is it doneH (X) = F ( xi )Most general binary function:H (X) xi concave0The transformation is to a submodular pair-wise MRF, hence optimization globally optimal[slide credits: Kohli]

Higher order to QuadraticStart with Pn Potts model:{0 if all xi = 0C1 otherwisef(x) =x {0,1}nxi = 0a=0f(x) = 0 xi > 0a=1f(x) = C1 [slide credits: Kohli]

Higher order to Quadraticmin f(x)min C1a + C1 (1-a) xix=x,a {0,1}Higher Order Function Quadratic Submodular Function xi123C1C1xi[slide credits: Kohli]

Higher order to Quadraticmin f(x)min C1a + C1 (1-a) xix=x,a {0,1}Higher Order Submodular Function Quadratic Submodular Function xi123C1C1xia=1a=0Lower envelope of concave functions is concave[slide credits: Kohli]

SummarySubmodular F2

F3 and beyond: minimization trickNon submodularQPBO(P)

Beyond F3 Robust HOP

What do you make out of this title?? NOTHING!So, Lets start with a little exampleWhat we are going to do in this lecture is to peek inside this black box and ask:What kind of energy functions we can work with:that have strong expressive poser AND can be minimizedWe will explore in this talk FAMILIES of functions;looking into their expressive power AND how to deal with them, i.e. how to minimize themMaking independent decision for each pixel does not get us too far We need stronger machinery MRF!!!Discrete minimization techniques:Belief propagation Loopy belief propagation, more recent works Tree-Reweighed belief propagation iterative algorithms for MAP estimation,

We want single snap solution!

xi \in T xi = 0xi \in S xi = 1Suppose Eij(0,0)=Eij(1,1) for all ijxi \in T xi = 0xi \in S xi = 1

The MOST IMPORTANT PART:cut partition = labelingCut weight = energy of labeling

Take a look at the slide (top down):Energy Graph partition=assignment cut weight = energy of assignment

This is the key point of this lecture!!!!State the obvious => MAP assignment min-cut/max-flow in poly time!Suppose Eij(0,0)=Eij(1,1) for all ijxi \in T xi = 0xi \in S xi = 1

The MOST IMPORTANT PART:cut partition = labelingCut weight = energy of labeling

Take a look at the slide (top down):Energy Graph partition=assignment cut weight = energy of assignment

This is the key point of this lecture!!!!State the obvious => MAP assignment min-cut/max-flow in poly time!****If I were to give this talk 5 years ago**Some insights into qpbo:- works when data term is dominant- works when there are not too many supermodular termsminimization trick if energy is submodular the trick will generate submodular terms!************Problems that are known to be easy- submodular- inference in treesAre indeed emerge as an easy problems with graph-cut formulation