View
231
Download
3
Category
Preview:
Citation preview
Video Modeling(Basics: Camera Models and Motion Models)
Two-Dimensional Motion Estimation(Fundamentals & Basic Techniques)
Chapter 5 (parts) and Chapter 6
2
Outline (chapter 5)
• Camera projection• 3D motion • Projection of 3-D motion• 2D motion due to rigid object motion
– Projective mapping
• Approximation of projective mapping– Affine model– Bilinear model
3
Pinhole Camera
Cameracenter
Imageplane
2-Dimage
3-Dpoint
The image of an object is reversed from its 3-D position. The object appears smaller when it is farther away.
4
Pinhole Camera Model:Perspective Projection
ZyxZYFy
ZXFx
ZY
Fy
ZX
Fx
torelatedinversely are ,
, ,
All points in this ray will have the same image
5
Approximate Model:Orthographic Projection
images) cmicroscopi i.e. Z,Ffor is Yy and X(xobject. theof distance the tocompared
small isobject in theation withdepth vari theas long as used beCan const whenF/Z with icorthograph scaled sY,y sX,actually x
, ,)(far very isobject When the
ZZYyXx
Z
6
Rigid Object Motion
zyxzyx TTT ,,:;,,:][;)]([':centerobject theon wrp. translatiandRotation
TRCTCXRX
7
Rigid Object Motion
],,[ zyx TTTT
zyxzyx TTT ,,:;,,:][;)]([':centerobject theon wrp. translatiandRotation
TRCTCXRX
z
y
x
TTT
ZYX
rrrrrrrrr
ZYX
987
654
321
'''
8
Flexible Object Motion
• Two ways to describe– Decompose into multiple, but connected rigid sub-objects– Global motion plus local motion in sub-objects– Ex. Human body consists of many parts each undergo a rigid
motion– Elastic motion can’t easily decomposed– Mesh models
9
3-D Motion -> 2-D Motion
2-D MV
3-D MV
Sample (2D) Motion Field
11
Occlusion Effect
Motion is undefined in occluded regions
12
Typical Camera Motions
13
2-D Motion Corresponding to Camera Motion
Camera zoom Camera rotation around Z-axis (roll)
14
2-D Motion Corresponding to Rigid Object Motion
• General case:
• Projective mapping:(if not planar – planar patches)
FTZFryrxrFTZFryrxr
Fy
FTZFryrxrFTZFryrxrFx
TTT
ZYX
rrrrrrrrr
ZYX
z
y
z
x
z
y
x
)()(
'
)()('
'''
987
654
987
321
Projection ePerspectiv
987
654
321
ycxcybxbby
ycxcyaxaax
21
210
21
210
1',
1'
:c)bYaX(Z object When the planar is surface
15
Projective Mapping
Two features of projective mapping:• Chirping: increasing perceived spatial frequency for far away objects• Converging (Keystone): parallel lines converge in distance
16
Affine and Bilinear Model
• Approximations to Projective Mapping– Affine (6 parameters):
• Good for mapping triangles to triangles
– Bilinear (8 parameters):
• Good for mapping blocks to quadrangles– Polynomial models (biquadratic – 12 parameters)
ybxbbyaxaa
yxdyxd
y
x
210
210
),(),(
xybybxbbxyayaxaa
yxdyxd
y
x
3210
3210
),(),(
17
Motion Field Corresponding toDifferent 2-D Motion Models
Translation
Bilinear Perspective
Affine
Outline (Chapter 6)二维运动估计
• Optical flow equation and ambiguity in motion estimation• General methodologies in motion estimation
– Motion representation– Motion estimation criterion– Optimization methods– Gradient descent methods
• Pixel-based motion estimation• Block-based motion estimation
– EBMA algorithm
19
2-D Motion vs. Optical Flow
On the left, a sphere is rotating under a constant ambient illumination, but the observed image does not change.
On the right, a point light source is rotating around a stationary sphere, causing the highlight point on the sphere to rotate.
• 2-D Motion: Projection of 3-D motion, depending on 3D object motion and projection operator• Optical flow: “Perceived” 2-D motion based on changes in image pattern, also depends on illumination and object surface texture
20
Optical Flow Equation
• When illumination condition is unknown, the best one can do it to estimate optical flow.
• Constant intensity assumption -> Optical flow equation
0or 0or 0
:equation flow optical thehave we two,above theCompare
),,(),,(
:expansion sTaylor' using But,
),,(),,(:"assumptionintensity constant "Under
ttv
yv
xd
td
yd
x
dy
dy
dx
tyxdtdydx
tyxdtdydx
Tyxtyx
tyxtyx
tyx
v
21
Optical Flow Equation
),,( ofector gradient v spatial theis ],[
vector velocity theis )v, ( where
0or 0
equation flow optic isThat
y
tyxyx
vtt
vy
vx
TT
x
Tyx
v
22
Ambiguities in Motion Estimation
• Optical flow equation only constrains the flow vector in the gradient direction
• The flow vector in the tangent direction ( ) is under-determined
• In regions with constant brightness ( ), the flow is indeterminate -> Motion estimation is unreliable in regions with flat texture, more reliable near edges
nv
0
0
tv
vv
n
ttnn
eev
tv
23
Ambiguities in Motion Estimation
nv
0
tv
• The flow vector in the tangent direction ( Vt ) is under-determined (aperture problem)
24
Ambiguities in Motion Estimation
• The flow vector in the tangent direction ( Vt ) is under-determined (aperture problem)
25
General Considerations for Motion Estimation
• Two categories of approaches:– Feature based (more often used in object tracking, 3D
reconstruction from 2D)– Intensity based (based on constant intensity assumption)
(more often used for motion compensated prediction, required in video coding, frame interpolation) -> Our focus
• Three important questions– How to represent the motion field?– What criteria to use to estimate motion parameters?– How to search motion parameters?
26
Motion Representation
Global:Entire motion field is represented by a few global parameters
Pixel-based:One MV at each pixel, with some smoothness constraint between adjacent MVs.
Region-based:Entire frame is divided into regions, each region corresponding to an object or sub-object with consistent motion, represented by a few parameters.
Block-based:Entire frame is divided into blocks, and motion in each block is characterized by a few parameters.
Also mesh-based (flow of corners, approximated inside)
27
Notations
Anchor frame:
Target frame:
Motion parameters:
Motion vector at a pixel in the anchor frame:
Motion field:
Mapping function:
)(1 x
)(2 x
)(xdxaxd ),;(
a
xaxdxaxw ),;();(
TLaaa ],...,,[ 21a
Backward motion estimation
Forward motion estimation
28
Motion Estimation Criterion
• To minimize the displaced frame difference (DFD)
(MSE)Error SquareMean ,min)());(()(
(MAD) Difference AbsoluteMean ,min)());(()(
212
12
MSE-DFD
MAD-DFD
x
x
E
E
xaxdxa
xaxdxa
min)()();()()(
0)(
)()( small is if 0
2
121OF
121
12111
x
pT
T
tttyx
E
dt
ddt
dy
dx
xxaxdxa
d
xx
• To satisfy the optical flow equation
29
Motion Estimation Criterion
10 1 1 1 2 1
12
2
( ( ( )) ( ) ) ( ( ( ) ( )) ( ))
( ) ( )( ) ( )( )
( )( )( )( ) ( )
T
x x
x x x x
y
xx x
v x x y x tv
y tx y y
d x x x x x
Block size: 8*8 16*16
This is homework
30
Motion Estimation Criterion
• Two above methods fail– Constant intensity assumption is not true everywhere (say shadows)– Flat texture: many motions satisfy the assumption (ill-posed)
• Need to impose additional smoothness constraint using regularization technique (Important in pixel- and block-based representation)
– Penalty term for smoothness, measure motion flow differences between adjacent pixels
– use weights to include a priori knowledge of importance of smoothness
min)()(
);();()(
DFD
2
aa
aydaxdax y
ssDFD
Ns
EwEw
Ex
31
Relation Among Different Criteria
• OF criterion is good only if motion is small. • OF criterion can often yield closed-form solution as the objective
function is quadratic in MVs.• When the motion is not small, can iterate the solution based on
the OF criterion to satisfy the DFD criterion.• Bayesian criterion can be reduced to the DFD criterion plus
motion smoothness constraint• More in the textbook
32
Optimization Methods
• Exhaustive search – Typically used for the DFD criterion with p=1 (MAD)– Guarantees reaching the global optimal– Computation required may be unacceptable when number of
parameters to search simultaneously is large!– Fast search algorithms reach sub-optimal solution in shorter time
• Gradient-based search– Typically used for the DFD or OF criterion with p=2 (MSE)
• the gradient can often be calculated analytically• When used with the OF criterion, closed-form solution may be obtained
– Reaches the local optimal point closest to the initial solution• Multi-resolution search
– Search from coarse to fine resolution, faster than exhaustive search– Avoid being trapped into a local minimum
33
Gradient Descent Method
• Iteratively update the current estimate in the direction opposite the gradient direction.
• The solution depends on the initial condition. Reaches the local minimum closest to the initial condition
• Choice of step side:– Fixed step size: Step size must be small to avoid oscillation, requires many iterations– Steepest gradient descent (adjust step size optimally)
Not a good initial
A good initial
Appropriatestepsize
Stepsize too big
34
Newton’s Method
• Newton’s method
– Converges faster than 1st order method (I.e. requires fewer number of iterations to reach convergence)
– Requires more calculation in each iteration– More prone to noise (gradient calculation is subject to noise, more so with 2nd order
than with 1st order)– May not converge if \alpha >=1. Should choose \alpha appropriate to reach a good
compromise between guaranteeing convergence and the convergence rate.
35
Newton-Raphson Method
• Newton-Ralphson method – Approximate 2nd order gradient with product of 1st order gradients– Applicable when the objective function is a sum of squared errors– Only needs to calculate 1st order gradients, yet converge at a rate similar to
Newton’s method.
Recommended