Mixture of trees model: Face Detection, Pose Estimation and

Landmark Localization

Presenter: Zhang Li

Problem

• Give an image, detect the face, pose estimation and the landmark points on each face

Existing works•separately handle the tasks

• Face detection: viola-Jones, Adaboost with LBP

• Pose estimation: Morphable model, 2D view based

• Facial Landmark: AAM, ASM, Flandmark(Deformable Part Models )

This work (CVPR2012)

• A unified model for face detection, pose estimation, and landmark estimation.

• Based on a mixtures of trees with a shared pool of parts

• Use global mixtures to capture topological changes

• Open source, result is par to commercial software, while training is based on hundreds of images

• May helpful for my building digital double project

Mixture of Trees Model • Generic model, can use for many tasks, such as object detection and human

tracking

Works lists on this area:1. Mixtures of Trees for Object Recognition, CVPR 20012. Human Tracking with Mixtures of Trees, ICCV 20013. Discriminative Mixture-of-Templates for Viewpoint Classification, ECCV20103. Articulated pose estimation with flexible mixtures of parts, CVPR20114. Face Detection, Pose Estimation, and Landmark Localization in the wild,

CVPR2012…..

Mixture of Trees Model

• Generic model, can use for many tasks, such as object detection and human tracking

Works lists on this area:1. Mixtures of Trees for Object Recognition, CVPR 2001 2. Human Tracking with Mixtures of Trees, ICCV 20013. Discriminative Mixture-of-Templates for Viewpoint Classification, ECCV20103. Articulated pose estimation with flexible mixtures of parts, CVPR20114. Face Detection, Pose Estimation, and Landmark Localization in the wild,

CVPR2012…..

To introduce the model

1. An object is s a collection of K primitives

Modeling with mixtures of trees

2. Primitive: a vector representing its configuration(e.g., the position in the image)

3. Given an image, the object detector will give a set of candidate of each primitive

Goal: build an assembly by choosing an element from each candidate set, so that the resulting set of primitives satisfies some global constraints.

Modeling with mixtures of trees

brute search, maximize

time consuming, time complexity

M is the number of candidate for each primitive

4. Instead, build a tree structure of K primitives

maximize

: root of the tree : parent of

MAP estimation on training data

Single tree to Mixtures of trees

Why this? Same to Gaussian to Multiple Gaussian

occlusions, variations in aspect or failures of the local detectors.

Therefore,

What set S of primitives consist of objects, therefore, in total, there will be components

Learning required:

and

:the weight of configuration as structure S

Mixtures of trees to shared structure

Why this?

Exploit structure is computationally expensive

Instead, use a seed to generate to approximate or some existing tree templates

A generating tree(seed) T : direct tree with K primitives, Then for each structure S, then

denote the event of this primitives belonging to the S

Grouping using mixtures of tress

Goal: localize an object in an image

maximize

Perform search on tree T

We select not only the best primitives to choose from the children’s candidate sets, but also the edges to be included in the tree(which parts constitute an object instance)

To see the application on face, refer to their paper

Mixture of Trees Model

• Generic model, can use for many tasks, such as object detection and human tracking

Works lists on this area:1. Mixtures of Trees for Object Recognition, CVPR 20012. Human Tracking with Mixtures of Trees, ICCV 20013. Discriminative Mixture-of-Templates for Viewpoint Classification, ECCV20103. Articulated pose estimation with flexible mixtures of parts, CVPR20114. Face Detection, Pose Estimation, and Landmark Localization in the wild,

CVPR2012…..

mixture-of-trees model

Prior Input: topological changes due to viewpoints, note no closed loops maintaining the tree property

How to modelEach facial landmark: as a part, similar to primitives

• We write each tree Tm =(Vm,Em) as a linearly-parameterized ,where m indicates a mixture and .

• I : image, and li = (xi, yi) : the pixel location of part I (the ith facial landmark).

• We score a configuration of parts

Meaning: the similarity of the input image I with facial landmarks positions as L under the m-th topology

: a scalar bias associated with viewpoint mixture m

Tree structured part model

Meaning: sums the appearance similarity for placing a template for part i, under the m-th topology, at location li.

Meaning: sums the mixture-specific spatial arrangement of parts L

: Local feature representation at location li

Shape modelthe shape model can be rewritten

• : re-parameterizations of the shape model (a, b, c, d), similar to AAM and ASM distance• : a block sparse precision matrix, with non-zero entries corresponding to pairs of parts i, j connected in Em.

Optimization• Inference corresponds to maximizing S(I, L,m) in Eqn.1 over L and

m:

• Since each mixture Tm =(Vm,Em) is a tree, the inner maximization can be done efficiently with dynamic programming.

Learning• Given labeled positive examples {In,Ln,mn} and negative examples

{In}, they will define a structured prediction objective function similar to one proposed in [41].

Rewrite, zn = {Ln,mn}.• Concatenating Eqn1’s parameters into a single vector

Concatenate and {a, b, c ,d } in to From (1), we know it is linear to and {a, b ,c ,d}

[41] Y. Yang and D. Ramanan. Articulated pose estimation using flexible mixtures of parts. In CVPR 2011.

Learning, max-margin(SVM)• Now we can learn a model of the form:

• The objective function penalizes violations of these constraints using slack variables

• write K for the indices of the quadratic spring terms (a, c) in parameter vector .

Experimental Results

Dataset

•CMU MultiPIE•annotated face in-the-wild (AFW) (from Flickr images)

Dataset

Sharing

•We explore 4 levels of sharing, denoting each model with the number of distinct templates encoded.▫Share-99 (i.e. fully shared model)▫Share-146▫Share-622▫Independent-1050 (i.e. independent model)

In-house baselines

•We define Multi.HoG to be rigid, multiview HoG template detectors, trained on the same data as our models.

•We define Star Model to be equivalent to Share-99 but defined using a “star” connectivity graph, where all parts are directly connected to a root nose part.

Face detection on AFW testset

[22] Z. Kalal, J. Matas, and K. Mikolajczyk. Weighted sampling for large-scale boosting. In BMVC 2008.

Pose estimation

Landmark localization

Landmark localization

AFW image

Conclusion

•This model outperforms state-of-the-art methods, including large-scale commercial systems, on all three tasks under both constrained and in-the-wild environments.