STATISTICAL ANALYSIS OF SHAPES OF 2D CURVES, 3DCURVES & FACIAL SURFACES

Anuj SrivastavaDepartment of Statistics

Florida State University

This research supported in part by ARO and NSF.

Presented at IMA Workshop on Shape Spaces, April 4th, 2006.

COLLABORATORS

• Mathematics: Eric Klassen, Washington Mio, Dave Wilson (Univ of

Florida)

• Computer Science: Mohamed Daoudi (ENIC, France), Chafik Samir

(grad, ENIC, France), Aastha Jain (undergrad, IIT Delhi)

• Statistics: Dave Kaziska (AFIT), Sanjay Saini (grad)

• Electrical Engineering: Shantanu Joshi (grad)

• Neuroscience: Zhaohua Ding and Chunming Li (Vanderbilt Inst. of

Imag. Sc.)

TENETS OF GRENANDER’S PATTERN THEORY

General Pattern Theory, Oxford University Press, 1993, In Introduction:

1. Create representations in terms of algebraic systems with probabilistic

superstructures.

What probabilistic superstructures should we have on our shape spaces?

2. Analyze structures from the perspective of ....statistical inference.

How can we generate inferences on such nonlinear, high-dimensional

space?

3. Apply ... to particular applications.

Are we there yet?

TOPICS COVERED

• Closed Curves in R2

– Geodesics Using Shooting Method

– Statistics, Bayesian Analysis

– Stochastic Processes on Shape Spaces

• Closed Curves in R3

– Geodesics Using Path-Straightening Method

• Shapes of Facial Surfaces

– Representations Using Facial Curves

– Face Recognition

• Summary

PAST WORK: PLANAR CURVES

1. Representations of curves: Assuming arc-length parametrization.

0 1 2 3 4 5 6 7−2

−1

0

1

2

3

4

5

6

7

Original Shape Angle Function θ

0 1 2 3 4 5 6 7−30

−20

−10

0

10

20

30

0 1 2 3 4 5 6 7−1.5

−1

−0.5

0

0.5

1

1.5

2

Curvature Function Coordinate Functions

C = {θ| 1

2π

Z 2π

0

θ(s)ds = π,

Z 2π

0

cos(θ(s))ds = 0,

Z 2π

0

sin(θ(s))ds = 0}

2. Shape-preserving transformations:

Automatically: Rotation, translation, and scaling.

Manually: Placement of origin, i.e. registration group is S1.

0 10 20 30 40 50 60 70 80 90 100−2

−1

0

1

2

3

4

5

6

7

8

0 10 20 30 40 50 60 70 80 90 100−1

0

1

2

3

4

5

6

7

Shape space S = C/S1.

3. Metric: L2 metric on angle functions

For g, h ∈ Tθ(S), we define:

〈g, h〉 =∫ 2π

0

g(s)h(s)ds .

4. Geodesics: Constructed numerically using shooting method. Paths

correspond to minimum bending energy between shapes.

0 2 4 6 8 10 12

0

1

2

0 2 4 6 8 10 12

0

1

2

5. References : Klassen et al., (Asilomar Conference (2002), EMMCVPR

(2003), and IEEE PAMI (March 2004)).

PAST WORK: ELASTIC PLANAR CURVES

1. Representations: Shapes are now represented by a pair of functions:

log-speed (φ) and angle (θ). eφ(s) provides speed of traversal along the

curve.

∫exp(φ)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

C = {(φ, θ)| 1

2π

Z 2π

0

θ(s)eφ(s)ds = π,

Z 2π

0

cos(θ(s))eφ(s)ds = 0,

Z 2π

0

sin(θ(s))eφ(s)ds = 0}

2. Shape-preserving transformations:

Automatic: Rotation, translation, and scaling.

Manually: (i) Placement of origin. (ii) Variable-speed parametrization, i.e.

re-parametrization group D: for γ ∈ D,

(φ, θ) ◦ γ = (φ ◦ γ + log(γ′), θ ◦ γ) .Shape space S = C/(S1 ×D).

Dynamic programming is used to match shapes.

3. Metric: L2 metric on angle and log-speed functions

For (g1, h1), (g2, h2) ∈ T(φ,θ)(S), we define:

〈(g1, h1, (g2, h2)〉 = a

∫ 2π

0

g1(s)g2(s)eφ(s)ds+b∫ 2π

0

h1(s)h2(s)eφ(s)ds .

4. Geodesics: Constructed numerically using shooting method. Paths

correspond to optimal bending and stretching/compression between

shapes.

Top row: non-elastic representation. Bottom row: elastic representation.

5. References : Mio et al., (CVPR 2004, IJCV in review).

SHAPE CLUSTERING

613 21

2344

7 14 24 25 36 39

918

2242

10 16 19 30 3345 47 48

1117

28 31

34 37 41 49

8 1526

3240

434612 20 27 29 35 38

50

1 2 3 4 5

Minimization of cumulative dispersion within classes using Metropolis-based

simulated annealing

References : Joshi et al., (SSP 2003, ECCV 2004, IEEE PAMI April 2005).

STATISTICS OF ELASTIC SHAPES

Examples of observed shapes

Intrinsic definition of mean shape: d(·, ·) denotes geodesic lengths.

µ = argminθ∈S

∫Sd(θ, φ)2f(φ)dφ .

TANGENT PRINCIPAL COMPONENTS (TPCA)

• Let Tµ(S) be the space of vectors tangent to S at µ ∈ S . Geodesics

are used to map shapes from S to Tµ(S).

• S is a nonlinear manifold but Tµ(S) is a vector space. Derive probability

models in this space. For example, use PCA to reduce dimension and

impose models on coefficients.

PROBABILITY MODELS ON TPCs

1. Non-parametric models:

• Let f(1)j , j = 1, . . . , d be the density estimate of aj , the jth TPC of

the shape α.

• Assuming independence of TPCs, we obtain:

f (1)(α) =∏dj=1 fj(aj)

2. Gaussian Models:

• Let Σ ∈ Rd×d be the diagonal matrix in SVD of K .

• The Gaussian model is given as, f (2)(α) =∏dj=1 h(aj ; 0,Σjj),

where h(y; z, σ2) ≡ 1√2πσ2 exp(−(y − z)2/(2σ2))

3. Gaussian Mixture models:

Define the mixture density

f (3)j(α) =

∏dj=1

(∑Kk=1 pkh(aj ; zk, σ

2k)

), and

∑k pk = 1.

Estimate f (3) using EM algorithm

• Density Estimates:

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40

0.5

1

1.5

2

2.5

−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40

0.5

1

1.5

2

2.5

−0.6 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.30

1

2

3

4

5

6

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.20

1

2

3

4

5

6

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40

0.5

1

1.5

2

2.5

3

3.5

4

4.5

−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50

0.5

1

1.5

2

2.5

3

3.5

4

−0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50

1

2

3

4

5

6

−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40

0.5

1

1.5

2

2.5

3

3.5

4

−0.2 −0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.2 0.250

1

2

3

4

5

6

Dogs Pears Mugs

• Likelihood Ratio Tests: Gaussian mixture outperforms others.

f(1)(α)f(3)(α)

5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

f(3)(α)f(2)(α)

5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

• References : Srivastava et al. (PAMI, April 2005, ACCV 2006).

BAYESIAN SHAPE ESTIMATION

MAP estimate of a closed curve, under a given prior model, by minimizing:

Etotal(α, I) = Eimage(α, I) + Esmooth(α) + Eprior(θ) .

Image I GVF of I Prior Mean

Evolution of curve under a gradient flow.

Reference: Joshi et al. (ECCV Workshop, 2006)

BAYESIAN SHAPE COMPLETION

Top row: Given images; Middle Row: Estimated shape using our approach;

Bottom row: Human drawing

EXTRACTION OF CARDIAC CURVES IN ULTRASOUND

(With Dave Wilson, UF)

ECG images of heart with expert generated contours for cardiac layers

overlaid.

Simply compute a geodesic path between the given shapes and overlay

them on the intermediate images.

References : Joshi et al. (IPMI, 2005).

Use the Bayesian approach to improve upon the estimation.

References : Joshi et al. (ECCV Workshop, 2006).

BAYESIAN SHAPE MODEL SELECTION

Approach: Shape is considered as nuisance variable in recognition

(Grenander et al. IEEE IT 2001):

....

...

0

5

10

15

20

25

30

35

40

models

E total

q1 q

3 q

5 q

7 q

9 q

4r q

6r q

8r q

2r

CYCLOSTATIONARY PROCESSES ON SHAPE SPACES

• Representation of human gait as a stochastic process X(t) on C.

• Cyclo-stationary process: statistics are periodic

• Metric for comparing gaits:

dp(X,Y ) = argminκ∈[0,τy ],ψ

(∫ τx

0

d(E[X(t)], E[Y (κ+ ψ(t))])2dt),

– E[·] is the expectation operator.

– κ, ψ are used to register the two gait cycles. ψ is solved using dynamic

programming.

– d(·, ·)2 is the elastic geodesic distance.

– Detection of gait cycles is automatic.

• Experimental Results: 26 subjects, different test and training data

i nearest neighbor 1 2 3 4 5 6 7 8 9 10

Elastic Shape Model 17 20 21 21 21 22 22 23 24 24

Mean-Shape Approach 13 14 16 19 19 20 20 20 20 22

Landmark-Based Approach 10 11 14 17 19 19 20 20 21 22

• References: Kaziska et al (PhD Thesis 2005, ECCV 2006).

3D FACE RECOGNITION

• We want to statistically analyze shapes of facial surfaces, i.e. define

geodesics, means, covariances, etc.

• Not as simple as curves, as there is no natural ordering of points.

• We chose a specific parametrization of facial surfaces.

3D FACE RECOGNITION USING FACIAL CURVES

• Assume that gaze direction is aligned with the z-direction. Let F be the

depth function on a facial surface.

• Let cλ be the level-curve of F for value λ. It has a shape component

and other variables (scale, rotation, etc).

• A surface S is represented by a path, parameterized by λ, on C(ignoring level curves that are not simple, closed curves).

• Examples:

• Metric: d(S1, S2) =∫λd(c1λ, c

2λ)dλ.

• Recognition rates: ∼ 60 people, 360 scans

1 1.5 2 2.5 3 3.5 4 4.5 530

40

50

60

70

80

90

Facial expressions per class used in training data sets

Rec

ogni

tion

Rat

e %

C5EucC4EucC53ucC2EucC1Euc

1 1.5 2 2.5 3 3.5 4 4.5 530

40

50

60

70

80

90

100

Facial expressions per class used in training data sets

Rec

ogni

tion

Rat

e %

C5GeoC4GeoC3GeoC2GeoC1Geo

1 1.5 2 2.5 3 3.5 4 4.5 560

65

70

75

80

85

90

Number of facial curves

Rec

ogni

tion

Rat

e %

C42Geo

C42Euc

• References: Samir et al (ICASSP 2006, PAMI in review).

INVARIANT FACIAL CURVES

• Define a function F (x) = d(x, p) where p is the tip of nose, and d(·, ·)is the geodesic distance on a facial surface. (Bronstein et al.IJCV 2005).

• Define facial curves to be the level curves of F .

• Compare surfaces by comparing their corresponding facial curves.

Facial curves are now closed curves in R3.

REPRESENTATION OF CLOSED CURVES in R3

• Let p : [0, 2π) �→ R3 be a C1-curve of length 2π,

parameterized by the arc length (can be relaxed as earlier).

• For v(s) ≡ p(s) ∈ R3, we have ‖v(s)‖ = 1 for all s ∈ [0, 2π).

• The function v is called the direction function of p and itself can be

viewed as a curve on the unit sphere S2, i.e. v : [0, 2π) �→ S

2.

REPRESENTATION OF CLOSED CURVES

• P be the set of all continuous maps from [0, 2π) → S2.

• Since we are interested in the shapes of closed curves, and we establish

that subset as follows. Define a map φ : P �→ R3 by

φ(v) =∫ 2π

0v(s)ds, and define

C = φ−1(0) ≡ {v ∈ P|φ(v) = 0} ⊂ P .

• Elements of C denote closed curves in R3.

• Inner product on Tv(C): for f, g ∈ Tv(C),

〈f, g〉 =∫ 2π

0

(f(s) · g(s))ds .

GEODESICS: A PATH STRAIGHTENING APPROACH

• Define Space: Let C be the space of all closed curves in R3. Let H be

the set of all paths in C, parameterized by t ∈ [0, 1].

• Initialize Path: For any two curves, denoted by v0 and v1, let α be a path

connecting them in C. That is, α : [0, 1] �→ C, such that α(0) = v0

and α(1) = v1.

• Path Energy: Let H0 be all such paths between v0 and v1. Define an

energy E on H0 whose critical point (gradient is zero) is a geodesic

path.

• Path-Straightening: Update α according to (negative) gradient of E until

convergence.

• Shape Space: Define a shape space S = C/T , and construct

geodesics in S .

PATH ENERGY AND ITS GRADIENT

• For a path α on H, it path energy is given by:

E(α) ≡ 12

∫ 1

0

⟨dα

dt,dα

dt

⟩dt .

(A critical point of E is a geodesic on C.)

• Goal: Our goal is to find the minimizer:

α = argminα∈H0

E(α) .

We will use a gradient approach to find this minimizer.

• To make H a Riemannian manifold, we use the Palais metric: for w1,

w2 ∈ Tα(H),

〈〈w1, w2〉〉 = 〈w1(0), w2(0)〉 +∫ 1

0

⟨Dw1

dt(t),

Dw2

dt(t)

⟩dt ,

Theorem 1 Let α : [0, 1] �→ C be a path such that α ∈ H0. Then, with

respect to the Palais metric:

1. The gradient of the energy function E on H is the vector field q along α

satisfying q(0) = 0 and Dqdt = dα

dt . That is, q is a covariant integral ofdαdt .

2. The gradient of the energy function E restricted to H0 is

w(t) = q(t) − tq(t), where q is the vector field defined in the previous

item, and q is the vector field obtained by parallel translating q(1)backwards along α.

At the critical point, we have q(t) = tq(t). Then, q(t) is covariantly

constant, q(t) = tq(t) is covariantly linear, and hence, dαdt is covariantly

constant.

References: Klassen et al (ECCV 2006, SIAM J. Computation, in review).

EXPERIMENTAL RESULTS

Experiment 1: Let the two curves of interest be:

p1(t) = (a cos(t), b sin(t), c√b2 − a2 sin2(t))

p2(t) = (a(1 + cos(t)), sin(t), 2 sin(t/2))

−1

−0.5

0

−1

−0.5

0

0.5

1

−0.15−0.1

−0.050

−1.5

−1

−0.5

0

−0.5

0

0.5

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1 1.5 2 2.5 3 3.5 4 4.5 5

3.45

3.5

3.55

3.6

Geodesic Path:

0

2

4

6

8

10

12

14

16

−2

0

2

−1

0

1

02

46

810

1214

16

−2

−1

0

1

2

−101

Experiment 2

0

0.2

0.4

0.6

0.8

1

1.2−0.2

0

0.2

0.4

0.6

0.8

1

−1

−0.5

0

−0.2

0

0.2

0.4

0.6

0.8

1

1.2 0

0.2

0.4

0.6

0.8

1

1.2

−0.8−0.6−0.4−0.2

00.2

0 2 4 6 8 10 12 1411

12

13

14

15

16

17

18

19

20

21

0 2 4 6 8 10 12 14 16

−2

−1

0

0 2 4 6 8 10 12 14 16

−1

0

1

2

FACIAL CURVES

Very preliminary:

−0.3

−0.2

−0.1

0

0.1

0.2

0.3−0.4

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.40.05

0.1

0.15

0.2

0.25

0.3

0.35

−0.4−0.2

00.2

0.4 −0.4

−0.2

0

0.2

0.4

0.1

0.15

0.2

0.25

0.3

0.35

0

2

4

6

8

10

12

−3

−2

−1

0

1

−1

0

1

SUMMARY

• Motivated of analysis of shapes of curves.

• Presented a representation of planar shapes that is invariant to rotation,

translation, and scaling. Showed examples of computing geodesic paths

in shape spaces.

• Presented examples of statistical analysis of shapes in R2, and imposed

probability models on shapes using TPCA.

• Demonstrated an application of probability models in Bayesian shape

extraction.

• Shapes of curves in R3: mentioned a path-straightening mechanism for

constructing geodesics.

• Used analysis of 3D curves to propose a method for studying shapes of

facial surfaces.