HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science

HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods

Albert Y. C. Chen1, Jason J. Corso1, and Le Wang2

1Dept. of Computer Science and Engineering2Dept. of Geography

University at Buffalo, The State University of New York

Image Labeling via Energy Minimization

Image labeling: assigning an object class (e.g. sky, water, tree, grass) to each pixel.

Energy minimization, or equivalently, posterior probability maximization, is the standard approach for solving image labeling problems.

Markov Random Fields or Conditional Random Fields, are the typical models used for energy minimization.

Vision and Perceptual Machines LabAlbert Y. C. Chen, Jason J. Corso, Le WangDepartment of Computer Science and Engineering

1

Typical Labeling Resultsversus the more ideal labeling by HOPS

Department of Computer Science and Engineering

Vision and Perceptual Machines LabAlbert Y. C. Chen, Jason J. Corso, Le Wang

Input Image to be labeledLabels using std. 1st orderMore Idea labels

2

Markov Random Field (MRF) in image labeling

Undirected-Graphs with cond. independent nodes. The labels of hidden nodes xi are chosen to

minimize the global energy. A hidden nodes xi is dependent only on other

hidden nodes within their Markov blanket (in practice, only first-order neighborhood is used).

Typical energy used:

(bias) (local evidence) (Markov blanket)

(E1: unary term) (E2: binary term)



3

Image Labeling via Energy Minimization – Previous Approaches

Simulated Annealing: Can converge to global optimum in theory, but is extremely slow in practice (several hours to several days).

Approximation Algorithms: Belief Propagation: energy reduction done by updating with

the messages passed along the nodes. Performance is to some degree un-expectable on loopy graphs.

Graph-Cuts: takes large leap in energy space, but is not guaranteed to converge on all energy functions.

Graph-Shifts: adaptive hierarchies are used to better represent the underlying data, while “shifts” are used to efficiently minimize the energy.



* For a detailed comparison of the performances, please refer to: Richard Szelisk et al., “A Comparative Study of EnergyMinimization Methods for Markov Random Fields with Smoothness-Based Priors”, PAMI 2008 4

Graph-Shifts: Its hierarchy and “shifts”Department of Computer Science and Engineering


1. Build the Adaptive Hierarchy2. A “SHIFT” occurs. It can happen at any level of the adaptive hierarchy.

3. Repeat until the overall energy is minimized.

5

Graph-Shifts: Its Effectiveness and Efficiency



Top: from J. J. Corso et al., IPMI 2007; Bottom: from J. J. Corso et al., CVPR 2008 6

Are MRF Blankets using first-order neighborhoods sufficient?

Algorithms with first-order neighborhood can be myopic, thus producing unnecessary label changes and even noisy/incorrect labeling results.

However, first-order neighborhood have been widely used by previous methods because: Higher-order neighborhood increases the inter-

nodal connectivity. Extending the neighborhood to the nth order

increases the comp. time by |N2|.|N3|. … |Ni|

HOPS, inspired by the Belief Propagation algorithm, approximates the energies of higher-order neighbors using the first order neighborhood.



7

HOPS (Higher Order Proxy NeighborhoodS)

Instead of calculating the binary energies between {μ, τi | τi: μ’s higher order neighbors} directly: The binary energy between νi (μ’s 1st order neighbors) and

their 1st order neighbors τj are passed to μ, to approximate the higher order binary energy between {μ, τi}.



8

HOPS (Higher Order Proxy NeighborhoodS) – why it works and how it works

When MRF with smoothness based prior is used and higher order neighborhood is requested, it means that: The label of a node μ and its 1st order neighbors ν are likely

to be equivalent. Thus, the approximated higher order binary energy of τ will

likely be the same as the direct calculation between {μ, τ}. First order binary energies are cached at each node n. Thus

when its neighboring node m is trying to approximate higher order energies, n would work as a proxy and pass its cached energy to m.

9



Natural Image Labeling ResultsDepartment of Computer Science and Engineering


Graph-Shifts using first order neighborhood*

Graph-Shifts with Higher Order Proxy NeighborhoodS

10* J. J. Corso et al., CVPR 2008

Discussion

11

Quality/Accuracy of the Labeling The number of small noisy labels is greatly reduced. Object boundaries are better followed in the labeled image The labeled images are much closer to what a human expert

would produce. The computation time for one “shift” have increased only

linearly. However, since redundant shifts are effectively avoided: The number of shifts required until convergence decreases

by an average of 60%. The overall convergence time is reduced by 30%.



What about other types of Energy function and problem sets?

Aerial photo and airborne LiDAR (light detection and ranging) - labeling results: HOPS gets approximately the

same accuracy rate yet

constantly converge 30% faster

than standard 1st order neighbor

ones. Other spatial constraints that relies on context of a larger

neighborhood can be added (such as shape information) into the model to achieve better results.

12



Conclusion

13

HOPS produces aesthetically and quantitatively better labeling results compared to those using standard first order neighbors.

HOPS estimates higher order energies in a recursive and cached manner, which induces little additional computational cost without increasing the node connectivity of the graph.

HOPS constantly converges 30% faster while used in the Graph-Shifts algorithm, since more context information is incorporated and redundant / incorrect label changes are more likely to be avoided.



Documents

HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science