Differential Motion Analysis (I)

Ying Wu

Electrical Engineering and Computer ScienceNorthwestern University, Evanston, IL 60208

yingwu@northwestern.edu

http://www.eecs.northwestern.edu/~yingwu

1 / 23

Outline

Optical FlowBrightness Constancy and Optical Flow ConstraintLucas-Kanade’s MethodHorn-Schunck’s MethodParametric FlowRobust Flow and Multi-frame Flow

Beyond Basic Optical FlowConsidering Lighting VariationsConsidering Appearance VariationsConsidering Spatial-Appearance Variations

2 / 23

Brightness Constancy and Optical Flow Optical flow: the apparent motion of the brightness pattern Optical flow 6= motion field Denote an image by I (x , y , t), and the velocity of a pixel

m = [x , y ]T is

vm = m = [vx , vy ]T =

[

dx/dt

dy/dt

]

Brightness constancy: the intensity of m keeps the sameduring dt, i.e.,

I (x + vxdt, y + vydt, t + dt) = I (x , y , t)

Optical flow constraint:

∂I

∂xvx +

∂I

∂yvy +

∂I

∂t= 0

i.e.

∇I · vm +∂I

∂t= 0

3 / 23

The Aperture Problem

For each pixel, one constraint equation, but two unknowns.

Normal flow

normalflow

optical constraint line

v

v

I

∆

x

y

Aperture problem: the motion along the directionperpendicular to the image gradient cannot be determined

p’

pC

C’

t

t

Other constraints are needed.

4 / 23

Lucas-Kanade’s Method Assume: a constant motion for a small image patch Ω. Define a weight function W (m),m ∈ Ω, for the pixels. Weighted LS formulation

minv

E =∑

m∈Ω

W 2(m)

(

∇I · v +∂I

∂t

)2

WLS solution:

v = (ATW2A)−1ATW2b

A =

∂I1∂x1

∂I1∂y1

......

∂IN∂xN

∂IN∂yN

, W = diag(W (m1), . . . ,W (mN))

v =

[

∂x

∂t,∂y

∂t

]T

= [vx , vy ]T , b = −[

∂I1∂t

, . . . , ∂IN∂t

]T

the intersection of all the flow constraint lines correspondingto the pixels in Ω.

5 / 23

Horn-Schunck’s Method

Assume: flow varies smoothly ←− global regularization

The measure of departure from smoothness can be written by:

es =

∫ ∫

(||∇vx ||2 + ||∇vy ||

2)dxdy

=

∫ ∫

(∂vx

∂x)2 + (

∂vx

∂y)2 + (

∂vy

∂x)2 + (

∂vy

∂y)2dxdy

The error of optical flow is:

ec =

∫ ∫

(∇I · vm +∂I

∂t)2dxdy

Objective function:

e = ec + λes

=

∫ ∫

(∇I · vm +∂I

∂t)2 + λ(||∇vx ||

2 + ||∇vy ||2)dxdy

6 / 23

Horn-Schunck’s Method

Fixed-point iteration:

vk+1x = vk

x −

(

∂I∂x

)

vkx +

(

∂I∂y

)

vky + ∂I

∂t

λ +(

∂I∂x

)2+(

∂I∂y

)2

∂I

∂x

vk+1y = vk

y −

(

∂I∂x

)

vkx +

(

∂I∂y

)

vky + ∂I

∂t

λ +(

∂I∂x

)2+(

∂I∂y

)2

∂I

∂y

Concisely, it is:vk+1 = vk − α(∇I )

In each iteration, the new optical flow field is constrained byits local average and the optical flow constraints.

7 / 23

Parametric Flow: Affine flow

Affine model is under two assumptions: planar surface orthographic projection

We can write a 3D plane by Z = AX + BY + C . Then wehave the 6-parameter affine flow model:

[

vx

vy

]

=

[

a1 a2

a3 a4

] [

x

y

]

+

[

a5

a6

]

In this case, the flow can be determined by at least 3 points.

8 / 23

Parametric Flow: Quadratic flow

Quadratic model is under two assumptions: planar surface perspective projection

Under perspective projection, a plane can be written as

1

Z=

1

C−

A

CX −

B

CY

So, we have

vx = a1 + a2x + a3y + a7x2 + a8xy

vy = a4 + a5x + a6y + a7xy + a8y2

In this case, if we know at least 4 points on a planar object,we can also a1, . . . , a8.

9 / 23

Parametric Flow: Parametric flow fitting LS formulation

minθ

∑

Ω

||I (x + vx(Θ)dt, y + vy (Θ)dt, t + dt)− I (x , y , t)||2

or

minθ

∑

Ω

[

∇ITv(Θ) + It

]2

denote by ∇Iθ = ∇IT∇v(Θ),

minθ

∑

Ω

[

∇ITθ Θ + It

]2

Easy to figure out the LS solution.

10 / 23

Exercises

Exercise 1: Recovering rotationAssume the motion is a pure rotation, i.e.,

[

x2

y2

]

= R(θ)

[

x1

y1

]

=

[

cos θ − sin θsin θ cos θ

] [

x1

y1

]

minθ

∑

[

I (R(θ)

[

x1

y1

]

, t + dt)− I (x , y , t)

]2

Exercise 2: Recovering 2D affine motion

[

vx

vy

]

=

[

a1 a2

a3 a4

] [

x

y

]

+

[

a5

a6

]

minA

∑

||∇ITv(A) + It ||2

11 / 23

Robust Flow Computation1

Motivation Brightness constancy

violation←− specular reflection

Spatial smoothnessviolation←− motion discontinuities

Outliers ruin LS estimation One solution: influence function ρ(x , σ)

Applying influence function to flow estimation

minv

∑

Ω

ρ(I (x , y , t)− I (x + vxdt, y + vydt, t + dt), σ)

minv

∑

Ω

ρc(∇ITv(θ) + It , σc) + λ[ρs(vx , σs) + ρs(vy , σs)]

1Michael Black and P. Anandan, “The Robust Estimation of Multiple Motions: Parametric and

Piecewise-Smooth Flow Fields”, CVIU, vol.63. no.1, pp.75-104, 199612 / 23

Multi-frame Optical Flow2

For a static scene, the flow induced by camera motion inmultiple frames lie in a low-dimensional subspace

Example: 3D planar scene under orthographic projection

[

vx

vy

]

=

[

a5 a1 a2

a6 a3 a4

]

1x

y

We have F frames and all have the same N points. Denote by[v ij

x vijy ]T the flow for the i-th point at the j-th frame, w.r.t. a

reference frame. Collect all the flows:

U =

v11x v21

x . . . vN1x

...... . . .

...v1Fx v2F

x . . . vNFx

=

a15 a1

1 a12

......

...aF5 aF

1 aF2

1 1 . . . 1x1 x2 . . . xN

y1 y2 . . . yN

2Michal Irani, “Multi-Frame Optical Flow Estimation Using Subspace Constraints”, ICCV’99

13 / 23

Multi-frame Optical Flow

And

V =

v11y v21

y . . . vN1y

...... . . .

...v1Fy v2F

y . . . vNFy

=

a16 a1

3 a14

......

...aF6 aF

3 aF4

1 1 . . . 1x1 x2 . . . xN

y1 y2 . . . yN

It is clear that

rank

[

UV

]

≤ 3

rank

[

U V]

≤ 6

14 / 23

Outline

Optical FlowBrightness Constancy and Optical Flow ConstraintLucas-Kanade’s MethodHorn-Schunck’s MethodParametric FlowRobust Flow and Multi-frame Flow

Beyond Basic Optical FlowConsidering Lighting VariationsConsidering Appearance VariationsConsidering Spatial-Appearance Variations

15 / 23

Considering lighting models

Brightness constancy assumption is too restrictive

Are there constraints for lighting?

For a pure Lambertain surface if no shadowing, then all images under varying illumination lie

in a 3-D subspace in RN

with shadowing, the dimension will be higher, but we maylearn it

The subspace can be learnt from a set of training images byPCA, so we have the basis B = [B1, B2, . . . ,Bm] (note:BTB = I).

Then the appearance of the template at t is modeled by

I (x , y , t) + BΛ, where Λ =

λ1...

λm

16 / 23

Considering lighting models3

therefore, we have

E (Θ,Λ) =∑

Ω

||I (x + vx(Θ)dt, y + vy (Θ)dt, t + dt)− I (x , y , t)− BΛ||2

or E (Θ,Λ) =∑

Ω

||∇Tv(Θ) + It − BΛ||2

or E (Θ,Λ) =∑

Ω

||∇Tθ Θ + It − BΛ||2

denote by ∇ITθ = M, we have

[

M −B]

[

ΘΛ

]

= −It

3Gregory Hager and Peter Belhumeur, “Real-Time Tracking of Image Regions with Changes in Geometry and

Illumination”, CVPR’96

17 / 23

Considering lighting models

So, we have

[

ΘΛ

]

=[

M −B]†

It

= −

[

MTM −MTB−BTM BTB

]−1 [MT

−BT

]

It

easy to see

Θ = −[

MT (I− BBT )M]−1

MT (I + BBT )It

18 / 23

Considering appearance variations

In-class appearance variations

Low-level matching −→ high-level matching

If we know the target, we may learn its appearance variations

We may build a classifier for matching

minv

S (I (u + vxdt, v + vydt, t + dt) : Λ) ,

where Λ are parameters of the classifier

E.g., using an SVM classifier

n∑

j=1

yjαjk(I, xj) + b

Let’s maximize the SVM matching score

maxu,v

n∑

j=1

yjαjk(I + uIx + vIy , xj)

19 / 23

Considering appearance variations4

Let use a 2nd order polynomial kernel k(x, xj) = (xTxj)2

so, we have

E (u, v) =n∑

j=1

yjαj

[

(I + uIx + vIy )Txj

]2

∂E

∂u=

∑

yjαj ITx xj(I + uIx + vIy )Txj = 0

∂E

∂v=

∑

yjαj ITy xj(I + uIx + vIy )Txj = 0

the solution is:[ ∑

αjyj(xTj Ix)

2∑

αjyj(xTj Ix)(xj Iy )

∑

αjyj(xTj Ix)(x

Tj Iy )

∑

αjyj(xTj Iy )2

] [

u

v

]

=

[

−∑

αjyj(xTj Ix)(x

Tj I )

−∑

αjyj(xTj Iy )(xT

j I )

]

4Shai Avidan, “Subset Selection for Efficient SVM Tracking”, CVPR’03

20 / 23

Spatial-appearance model (SAM)5

Denote by y = [x c(x)], where x is the location and c color Assume a Gaussian component be factorized

g(y; µk , Σk) = g(x; µks , Σks)g(c(x); µkc , Σkc)

For a pixel, the likelihood is a mixture

p(y|Θ) =K∑

k=1

pkg(y; µk , Σk)

Let’s use an affine motion here

T (x; at) =

[

a1t a2t

a3t a4t

]

x +

[

a5t

a6t

]

then, we have

p(T (y; at)|Θ) = p(T (x; at), c(T (x; at))|Θ)

=K∑

k=1

pkg(x;µks ,Σks)g(c(T (x; at));µkc ,Σkc)=

K∑

k=1

q(k, yi ; at)

5Ting Yu and Ying Wu, “Differential Tracking based on Spatial-Appearance Model (SAM)”, CVPR’06

21 / 23

Spatial-appearance model (SAM)

For an image region

E (at ; Θ) =∑

xi∈Ω

log p(T (yi ; at)|Θ) =∑

xi∈Ω

log

(

K∑

k=1

q(k , yi ; at)

)

our task is tomax

at

E (at ; Θ)

Solution: similar to the general EM algorithm

22 / 23

Spatial-appearance model (SAM)

23 / 23