Computer Vision Optical Flow Marc Pollefeys COMP 256 Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black,

ComputerVision

Optical Flow

Marc PollefeysCOMP 256

Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black, K. Toyama

ComputerVision

2

last week: polar rectification

ComputerVision

3

Last week: polar rectification

ComputerVision

4

Stereo matching

Optimal path(dynamic programming )

Similarity measure(SSD or NCC)

Constraints• epipolar

• ordering

• uniqueness

• disparity limit

• disparity gradient limit

Trade-off

• Matching cost (data)

• Discontinuities (prior)

(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

ComputerVision

5

Hierarchical stereo matchingD

ow

nsam

plin

g

(Gau

ssia

n p

yra

mid

)

Dis

pari

ty p

rop

ag

ati

on

Allows faster computation

Deals with large disparity ranges

(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

ComputerVision

6

Disparity map

image I(x,y) image I´(x´,y´)Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

ComputerVision

7

Example: reconstruct image from neighboring images

ComputerVision

8

Multi-view depth fusion

• Compute depth for every pixel of reference image– Triangulation– Use multiple views– Up- and down sequence– Use Kalman filter

(Koch, Pollefeys and Van Gool. ECCV‘98)

Allows to compute robust texture

ComputerVision

9

Real-time stereo on graphics hardware

• Computes Sum-of-Square-Differences• Hardware mip-map generation used to aggregate

results over support region• Trade-off between small and large support window

Yang and Pollefeys CVPR03

140M disparity hypothesis/sec on Radeon 9700pro140M disparity hypothesis/sec on Radeon 9700proe.g. 512x512x20disparities at 30Hze.g. 512x512x20disparities at 30Hz

Shape of a kernel for summing up 6 levels

ComputerVision

10

Sample Re-Projections

near far

ComputerVision

11

(1x1)

(1x1+2x2)

(1x1+2x2 +4x4+8x8)

(1x1+2x2 +4x4+8x8 +16x16)

Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass

video

ComputerVision

12

Cool ideas

• Space-time stereo (varying illumination, not shape)

ComputerVision

13

More on stereo …

The Middleburry Stereo Vision Research Pagehttp://cat.middlebury.edu/stereo/

Recommended reading

D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation.Microsoft Research Technical Report MSR-TR-2001-81, November 2001. PDF file (1.27 MB).

ComputerVision

14

Jan 16/18 - Introduction

Jan 23/25 Cameras Radiometry

Jan 30/Feb1 Sources & Shadows Color

Feb 6/8 Linear filters & edges Texture

Feb 13/15 Multi-View Geometry Stereo

Feb 20/22 Optical flow Project proposals

Feb27/Mar1 Affine SfM Projective SfM

Mar 6/8 Camera Calibration Silhouettes and Photoconsistency

Mar 13/15 Springbreak Springbreak

Mar 20/22 Segmentation Fitting

Mar 27/29 Prob. Segmentation Project Update

Apr 3/5 Tracking Tracking

Apr 10/12 Object Recognition Object Recognition

Apr 17/19 Range data Range data

Apr 24/26 Final project Final project

Tentative class schedule

ComputerVision

15

Optical Flow

• Brightness Constancy• The Aperture problem• Regularization• Lucas-Kanade• Coarse-to-fine• Parametric motion models• Direct depth• SSD tracking• Robust flow• Bayesian flow

ComputerVision

16

Optical Flow:Where do pixels move to?

ComputerVision

17

Motion is a basic cue

Motion can be the only cue for segmentation

ComputerVision

19

Motion is a basic cue

Even impoverished motion data can elicit a strong percept

ComputerVision

20

Applications

• tracking• structure from motion• motion segmentation• stabilization• compression• Mosaicing• …

ComputerVision

21

Optical Flow


ComputerVision

22

Definition of optical flow

OPTICAL FLOW = apparent motion of brightness patterns

Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image

ComputerVision

23

Caution required !

Two examples :

1. Uniform, rotating sphere

O.F. = 0

2. No motion, but changing lighting

O.F. 0

ComputerVision

24

Caution required !

ComputerVision

25

Mathematical formulation

I (x,y,t) = brightness at (x,y) at time t

Optical flow constraint equation :

0

t

I

dt

dy

y

I

dt

dx

x

I

dt

dI

),,(),,( tyxItttdt

dyyt

dt

dxxI

Brightness constancy assumption:

ComputerVision

26

Optical Flow


ComputerVision

27

The aperture problem

0 tyx IvIuI

1 equation in 2 unknowns

dt

dxu

dt

dyv

y

II x

y

II y

t

II t

ComputerVision

28


0

ComputerVision

29

ComputerVision

30


ComputerVision

31

Remarks

ComputerVision

32

Apparently an aperture problem

ComputerVision

33

What is Optic Flow, anyway?

• Estimate of observed projected motion field

• Not always well defined!• Compare:

– Motion Field (or Scene Flow)projection of 3-D motion field

– Normal Flowobserved tangent motion

– Optic Flowapparent motion of the brightness pattern(hopefully equal to motion field)

• Consider Barber pole illusion

ComputerVision

34

ComputerVision

35

Optical Flow


ComputerVision

36

Horn & Schunck algorithm

Additional smoothness constraint :

,))()(( 2222 dxdyvvuue yxyxs besides OF constraint equation term

,)( 2 dxdyIvIuIe tyxc

minimize es+ec

ComputerVision

37

Horn & Schunck

The Euler-Lagrange equations :

0

0

yx

yx

vvv

uuu

Fy

Fx

F

Fy

Fx

F

In our case ,

,)()()( 22222tyxyxyx IvIuIvvuuF

so the Euler-Lagrange equations are

,)(

,)(

ytyx

xtyx

IIvIuIv

IIvIuIu

2

2

2

2

yx

is the Laplacian operator

ComputerVision

38

Horn & Schunck

Remarks :

1. Coupled PDEs solved using iterative methods and finite differences

2. More than two frames allow a better estimation of It

3. Information spreads from corner-type patterns

,)(

,)(

ytyx

xtyx

IIvIuIvt

v

IIvIuIut

u

ComputerVision

39

ComputerVision

40

Horn & Schunck, remarks

1. Errors at boundaries

2. Example of regularisation (selection principle for the solution of illposed problems)

ComputerVision

41

Results of an enhanced system

ComputerVision

42

Structure from motion with OF

ComputerVision

43

Optical Flow


ComputerVision

44

02),(

02),(

tyxy

tyxx

IvIuIIdv

vudE

IvIuIIdu

vudE

ComputerVision

45

ComputerVision

46

ComputerVision

47

ComputerVision

48

ComputerVision

49

Optical Flow


ComputerVision

50

ComputerVision

51

ComputerVision

52

ComputerVision

53

Optical Flow


ComputerVision

54

ComputerVision

55

ComputerVision

56

ComputerVision

57

ComputerVision

58

ComputerVision

59

ComputerVision

60

ComputerVision

61

ComputerVision

62

ComputerVision

63

ComputerVision

64

ComputerVision

65

ComputerVision

66

ComputerVision

67

ComputerVision

68

ComputerVision

69

ComputerVision

70

ComputerVision

71

ComputerVision

72

ComputerVision

73

ComputerVision

74

ComputerVision

75

ComputerVision

76

ComputerVision

77

ComputerVision

78

ComputerVision

79

ComputerVision

80

ComputerVision

81

ComputerVision

82

ComputerVision

83

Optical Flow


ComputerVision

84

ComputerVision

85

ComputerVision

86

ComputerVision

87

Optical Flow


ComputerVision

88

ComputerVision

89

ComputerVision

90

ComputerVision

91

ComputerVision

92

ComputerVision

93

Optical Flow


ComputerVision

94

ComputerVision

95

ComputerVision

96

ComputerVision

97

ComputerVision

98

ComputerVision

99

ComputerVision

100

Optical Flow


ComputerVision

101

ComputerVision

102

Rhombus Displays

http://www.cs.huji.ac.il/~yweiss/Rhombus/

ComputerVision

103

ComputerVision

104

ComputerVision

105

ComputerVision

106

ComputerVision

107

ComputerVision

108

Documents

Computer Vision Optical Flow Marc Pollefeys COMP 256 Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black,