22
Optical Flow Estimation Using High Frame Rate Sequences Suk Hwan Lim and Abbas El Gamal Programmable Digital Camera Project Department of Electrical Engineering, Stanford University, CA 94305, USA ICIP 2001 1

Optical Flow Estimation Using High Frame Rate Sequences

  • Upload
    others

  • View
    10

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Optical Flow Estimation Using High Frame Rate Sequences

Optical Flow Estimation

Using High Frame Rate Sequences

Suk Hwan Lim and Abbas El Gamal

Programmable Digital Camera Project

Department of Electrical Engineering, Stanford University, CA 94305, USA

ICIP 2001 1

Page 2: Optical Flow Estimation Using High Frame Rate Sequences

Digital Imaging System Implementation

PC−Board

ASIC

MemoryADC &SensorImageCMOS

PC−Board

(camera−on−chip)digital camera

Single chip

CCD

Memory

AnalogProc &ADC

ASIC

• CMOS image sensor:

– Integration of camera functions with sensor on

same chip

– Low power consumption

– High frame rate imaging

ICIP 2001 2

Page 3: Optical Flow Estimation Using High Frame Rate Sequences

High Speed CMOS Image Sensor Examples

• Fossum et al. (VLSI symposium 1999):

– 1024×1024 array with 500 fps

– APS with 10µm×10µm pixel size

• Stevanovic et al. (ISSCC 2000):

– 256×256 array with 1024 fps

– APS with 30µm×30µm pixel size

• Kleinfelder et al. (ISSCC 2001):

– 352×288 array with 10,000 fps

– DPS with 9.4µm×9.4µm pixel size

– ADC on each pixel

ICIP 2001 3

Page 4: Optical Flow Estimation Using High Frame Rate Sequences

Motivation

• Exploit high speed imaging capability to improve

still and standard video rate imaging applications

– Dynamic range enhancement

– Motion blur-free capture

– Optical flow estimation

– Video stabilization

– Super-resolution

Integration of capture and processing on same chip

makes system implementation feasible

ICIP 2001 4

Page 5: Optical Flow Estimation Using High Frame Rate Sequences

Multiple Capture for Video/Data Enhancement

TimeHigh frame rate capture Standard frame rate output

+Output video

Application specificoutput data

Processing

• Operate the sensor at high frame rate

• Process high frame rate data on-chip

• Output video with any application specific data at

standard frame rate

ICIP 2001 5

Page 6: Optical Flow Estimation Using High Frame Rate Sequences

Optical Flow Estimation (OFE)

• Applications

– 3D motion and structure estimation

– Super-resolution

– Image restoration

• Accuracy is of primary concern

ICIP 2001 6

Page 7: Optical Flow Estimation Using High Frame Rate Sequences

Block Diagram of Our OFE Method

imager

High speed Accumulate

and refineframe rate

Optical flow

at standard

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

Adaptively change frame rate

EstimateOptical flow

(Lucas−Kanade)rate sequenceHigh frame

high frame rateOptical flow at

Output frames

Intermediate frames

�������������������������������������������������������

�������������������������������������������������������

ICIP 2001 7

Page 8: Optical Flow Estimation Using High Frame Rate Sequences

Effect of High Frame Rate on Optical Flow Estimation

• Advantages for gradient-based methods

– Brightness constancy assumption, i.e.

dI(x, y, t)

dt=

∂I

∂xvx +

∂I

∂yvy +

∂I

∂t= 0

becomes more valid with higher frame rate

– Less temporal aliasing

– Temporal derivatives better estimated

– Smaller kernel size needed

• Disadvantages

– Lower SNR

ICIP 2001 8

Page 9: Optical Flow Estimation Using High Frame Rate Sequences

Lucas-Kanade OFE Method

SmoothingGradient

Estimation

Construct

2x2 matrix

Solve linear

equation

[ ∑wI2

x

∑wIxIy∑

wIxIy∑

wI2y

] [vx

vy

]= −

[ ∑wIxIt∑wIyIt

]

• Ix,Iy and It are partial derivatives computed using

5-tap filters

• w(x, y) puts more weight to the center of

neighborhood (5×5)

ICIP 2001 9

Page 10: Optical Flow Estimation Using High Frame Rate Sequences

Accumulate and Refine

For i = 0, ..., OV :

������������������������������������������������

������������������������������������������������

������������������������������������������������

������������������������������������������������

���������������������������������

���������������������������������

given

frame0 frame i frame i+1

Accumulate(2)

Warp(3)

Refine(4)

Estimate OFE(1)

OV is the temporal oversampling ratio

ICIP 2001 10

Page 11: Optical Flow Estimation Using High Frame Rate Sequences

Experimental setup

ParametersMotion Sequence

GenerationTrue Optical Flow

Sequence @

Sequence @ high frame rate

standard frame rate

+

+

OFE error 1

OFE error 2

Standard

OFE

Our OFE

• Displacement can be controlled and are known

• Motion blur and noise added

• Effect of frame rate on image quality included

• Standard OFE implemented by Barron et. al

ICIP 2001 11

Page 12: Optical Flow Estimation Using High Frame Rate Sequences

Video Sequence Model

• The output charge from each pixel:

Q(m, n) =

∫ T

0

∫ ny0+Y

ny0

∫ mx0+X

mx0

j(x, y, t)dxdydt + N(m, n)

– (m, n) is the pixel index

– x0 and y0 are the pixel dimensions

– X and Y are the photodiode dimensions

– T is the exposure time

– j(x, y, t) A/cm2 is the photocurrent density

– N(m, n) is the noise charge

• Pixel intensity, I, proportional to Q(m, n)

ICIP 2001 12

Page 13: Optical Flow Estimation Using High Frame Rate Sequences

Synthetic Sequence Generation

Integrate &

SubsampleAdd noise QuantizeWarp

1. Warp a high resolution (1312 × 2000) image using

perspective warping

2. Integrate and subsample spatially (4 × 4) and

temporally (10)

3. Add readout noise and shot noise according to the

model

4. Quantize the sequence

ICIP 2001 13

Page 14: Optical Flow Estimation Using High Frame Rate Sequences

Example Original Scene and Optical Flow

Original scene Optical flow

ICIP 2001 14

Page 15: Optical Flow Estimation Using High Frame Rate Sequences

Experiment I

Compare standard Lucas-Kanade OFE and our OFE

Sequence @ 30 fps

Sequence @ 120 fps

Optical Flow Estimate @ 30fps

Optical Flow Estimate @ 30fps

Time

Time

(OV=4)

Our

(A)

(B)

Lucas−Kanade

Standard

OFE

OFE

Displacement < 4 pixels/frame at standard frame rate

ICIP 2001 15

Page 16: Optical Flow Estimation Using High Frame Rate Sequences

Result I

SceneAngular error

Our method (B)

DensityAngular errorDensity

Lucas−Kanade method (A)

1

2

3

4.43◦3.94◦4.56◦

55.0%

53.0%

53.5%

3.43◦2.91◦2.67◦

55.7%

53.4%

53.4%

• Higher accuracy achieved with our method

• More difference when brightness constancy does

not hold

• Temporal filters

– Our method: 2-tap

– Lucas-Kanade method: 5-tap

ICIP 2001 16

Page 17: Optical Flow Estimation Using High Frame Rate Sequences

Experiment II

Investigate accuracy gain for large displacements

Standard

Lucas−Kanade

Matching

Hierarchical

OFE

OFE

(OV=10)

Our OFE

(A)

(C)

(B)

Displacement < 10 pixels/frame at standard frame rate

ICIP 2001 17

Page 18: Optical Flow Estimation Using High Frame Rate Sequences

Result II

Hierarchical matching method

Angular error Density

Lucas−Kanade method

Our method (OV=10)

9.18◦4.72◦1.82◦

50.81%

100%

50.84%

• Standard Lucas-Kanade method deteriorates for

large displacements

• Hierarchical matching method has 100% density

but lower accuracy

• Our method works well for both small and large

displacements

ICIP 2001 18

Page 19: Optical Flow Estimation Using High Frame Rate Sequences

Experiment III

Investigate the effect of varying OV on accuracy

Our OFE

Our OFE

(OV=1)

Our OFE

(OV=2)

(OV=10)

ICIP 2001 19

Page 20: Optical Flow Estimation Using High Frame Rate Sequences

Result III

0 5 10 150

1

2

3

4

5

6

Ave

rag

e A

ng

ula

r E

rro

r(d

egre

es)

Oversampling factor(OV)

Temporal aliasing, temporal gradient estimation error,

failure in brightness constancy and sensor SNR are

affected by OV

ICIP 2001 20

Page 21: Optical Flow Estimation Using High Frame Rate Sequences

Hardware Complexity

Lucas−Kanade methodOur method

Memory (bytes)

Operations

12mn

190mnOV

16mn

105mn

• Assumptions:

– m × n image with oversampling factor of OV

– 5-tap spatial filter for gradient estimation and

smoothing

– 2-tap temporal filter for our method and 5-tap

for Lucas-Kanade method

• Note memory requirement is independent of OV

since our method is iterative

ICIP 2001 21

Page 22: Optical Flow Estimation Using High Frame Rate Sequences

Conclusion

• High frame rate and integration capabilities of

CMOS image sensors can be exploited to improve

the performance of video processing applications

• Developed a method for accurate optical flow

estimation using high frame rate sequences

• Demonstrated that our method

– obtains higher accuracy than OFE using

standard frame rate sequences

– works well for large displacements

– requires modest memory and computational

power since our method is iterative

ICIP 2001 22