Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
ECE 634 – Digital Video SystemsSpring 2019
Fengqing Maggie ZhuAssistant Professor of ECE
MSEE [email protected]
1
Motion Models
Outline
• Camera model and camera motion• Object motion• Models to represent motion
2
Why Study Motion?
• Motion creates spatio-temporal information• Understand 3D motion• Better object recognition, identification, segmentation• Sometimes the information IS the motion (ex: action
recognition)• Improve video quality through motion stabilization
• Objects themselves don’t change much during motion
• Remove temporal redundancies for compression• Motion-compensated temporal filtering (along motion
trajectories) to remove noise• Motion-compensated frame interpolation
3
Example Videos
• Akiyo: object motion• Bus: object + camera motion• City: camera motion
4
https://www.biomotionlab.ca/Demos/BMLwalker.html5
Reading Material
• A. M. Tekalp, Digital video processing, Prentice Hall, 2015
• Chapter 4.1,4.2
• Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing and Communications, Prentice Hall, 2002
• Chapter 5.1, 5.3.2, 5.5
6
Motion Detection
• Is an image region moving or stationary?• We will postpone this discussion until later
7
Motion Estimation
• Need accurate models for the motion• Understand the camera and its motion• Understand objects and their motion
• All we can see from the camera is the 2D projection of the 3D world
• This mapping is not unique! Many 3D-worlds can produce the same 2D image
• All our interpretations of the world must take place through this projection
8
Motion Estimation Applications
• Motion for interpreting 3D world requires an inverse mapping
• Motion for compression need not approximate physical motion; it is solely to reduce data rate
• Processing with motion is best if the true motion of image points can be obtained
Different applications/purposes of motion require different approaches for motion estimation
9
To Estimate 2D Motion We Need
• A motion model• An estimation criterion• A search strategy
10
11
Summary of Motion Models
• What causes 2D motion?• Camera motion• Object motion projected to 2D
• Camera model: 3D à 2D projection• Perspective projection vs. orthographic projection
• Models corresponding to typical camera motion and object motion
• Piece-wise projective mapping is a good model for projected rigid object motion
• Can be approximated by affine or bilinear functions• Affine functions can also characterize some global camera motions
• Ways to represent motion• Pixel-based, block-based, region-based, global, etc.
2D Motion
2D (or apparent) motion that is created by moving objects depends on 3 things:1. An image formation (or camera) model
• Perspective, orthographic, ..2. Motion model of a 3D object (rigid body with 3D
translation and rotation, 3D affine motion)3. Surface model of 3D object (planar, parabolic..)
12
Camera Models
• Orient the camera in 3D space• Basic camera model (pinhole)• Simpler orthographic model• More complex camera models exist
• Example: CAHV – characterizes practical camera system by its extrinsic and intrinsic parameters
13
14
Pinhole Camera
Cameracenter
Imageplane
2-Dimage
3-Dpoint
The image of an object is reversed from its 3-D position.
Perspective: The object appears smaller when it is farther away.
15
Pinhole Camera Model:Perspective Projection
ZyxZYFy
ZXFx
ZY
Fy
ZX
Fx
torelatedinversely are ,
,, ==Þ==
All points in this ray will have the same image
Z is depth;Distance fromcamera centerto object
Attributes of Perspective Projection
• Straight lines in 3D space are projected into straight lines in 2D space
• Parallel lines in 3D may not be parallel in 2D• Vanishing points
• A nonlinear model, due to division by depth Z
16
Vanishing Points
17
18
Simpler Model:Orthographic Projection
object. theof distance the tocomparedsmall isobject in theation withdepth vari theas long as used beCan
,)(far very isobject When the
YyXxZ
==¥®
CAHV camera model
• (extrinsic) Vector C: location of the pinhole• (extrinsic) Vector A: Camera axis (normal to the
image plane)• (intrinsic) Vector H:
• Horizontal axis of image plane• H-coordinate of optical center of image plane• Horizontal focal length (in pixels)
• (intrinsic) Vector V: same vertically
A number of other camera models available, all more sophisticated than pinhole model19
Translational Camera Motions in 3D
Track (x), Boom (y), Dolly (z)
20
Rotational Camera Motions in 3D
Tilt (x), Pan (y), Roll (z)
21
2D Motion
2D (or apparent) motion that is created by moving objects depends on 3 things:1. An image formation (or camera) model
• Perspective, orthographic, ..2. Motion model of a 3D object (rigid body with 3D
translation and rotation, 3D affine motion)3. Surface model of 3D object (planar, parabolic..)
22
Motion models for 3D objects
• 3D motion • Projection of 3-D motion on 2-D image plane• 2D motion caused by rigid object motion
• Projective mapping• Approximations to the 2D motion field
• Affine model• Bilinear model
23
Rigid Object 3-D Motion
• Translation vector T, defines how object translates in x,y,z; T=[Tx,Ty,Tz]’
• Rotation matrix R• Defines how object rotates, first in X, then in Y, then in Z• Defined by rotation angles:
• 6 parameters, valid for all points on object
24
zyx qqq ,,
Rotation Matrix
[R] is orthonormal, i.e. it satisfies
25
Linearization Approximation
• Assume rotation angles are small so that
26
Rigid Object 3-D Motion
27
zyxzyx TTT ,,:;,,:][;)]([':centerobject theon wrp. translatiandRotation
TRCTCXRX qqq++-=
Rigid vs Flexible Object
• An object is rigid, if all its points move with the same 3D motion parameters
• Two ways to describe flexible object• Decompose into multiple, but connected rigid sub-
objects• Global motion plus local motion in sub-objects• Ex. Human body consists of many parts each undergo a
rigid motion
28
29
3-D Motion à 2-D Motion
2-D MV
3-D MVRemember: Many 3D MV’smap to the same 2D MV.So the inverse problem isill-defined.
Sample 2-D Motion Field
A “needle” plot helps to visualize motion, bydescribing where to go in first frame to get the “matching” pixel values for the second frame
30
2D Motion Models
• How is the 2D motion represented in 2D• Each pixel can move on its own, but it’s often more
effective to describe how a region of pixels move –because remember, 2D motion is caused either by camera motion (which moves all pixels) or by objectmotion (which moves all the pixels on an object)
• So next, we look at how camera motion appears in 2D
• Take a point (X,Y,Z) in 3D, map it into its new coordinates (X’,Y’,Z’) based on camera motion, the apply perspective mapping to find relationship between (x,y) and (x’,y’)
31
32
2-D Motion Corresponding to Camera Motion
Camera zoom Camera rotation around Z-axis (roll)
Also Pan and tilt (with small angle approximations)
Camera Track And BoomTranslation: Tx , Ty
3D position of any point:
33
The imaged position: When Z is large-ish:
Camera Pan and TiltRotation angles: 𝜃y ,𝜃 x
34
When rotation angle are small:
[Rx] and [Ry] as defined earlier
[R]
Camera ZoomF and F’ are focal length before and after
is the zoom factor
35
The 2D motion field is
Camera RollRotation around the Z-axis, no change in depth
36
The 2D motion field is
Four Parameter Model
• Consider a camera that goes through translation, pan, tilt, zoom, and rotation in sequence
• With small angle approximation:
• A special case of the affine mapping, which generally has six parameters.
37
2-D Motion Corresponding to Rigid Object Motion• General case:
• Projective mapping:
38
FTZFryrxrFTZFryrxr
Fy
FTZFryrxrFTZFryrxrFx
TTT
ZYX
rrrrrrrrr
ZYX
z
y
z
x
z
y
x
+++
+++=
++++++
=
¾¾¾¾¾¾ ®¾
úúú
û
ù
êêê
ë
é+úúú
û
ù
êêê
ë
é
úúú
û
ù
êêê
ë
é=
úúú
û
ù
êêê
ë
é
)()(
'
)()('
'''
987
654
987
321
Projection ePerspectiv
987
654
321
ycxcybxbby
ycxcyaxaax
21
210
21
210
1',
1'
:c)bYaX(Zplanar is surfaceobject When the
++++
=++++
=
++=
(8 parameters)
More 2D motion models
• Projective mapping (8 parameters)• (3D rigid motion of planar surface using perspective
projection)• Affine
• (3D rigid motion of planar surface using orthographic projection)
• Bilinear• Translational
39
Projective Mapping
Two features of projective mapping:• Chirping: increasing perceived spatial frequency for far
away objects• Converging (Keystone): parallel lines converge in distance
40
41
Affine Model
• Polynomial approximation to projective mapping• 6 parameters
• Maps triangles to triangles: defined by motion vectors of the three corners
úû
ùêë
é++++
=úû
ùêë
éybxbbyaxaa
yxdyxd
y
x
210
210
),(),(
42
Affine Model
Affine model:• No chirping, no converging of parallel lines
43
Bilinear Model
• Another approximation to projective mapping• 8 parameters
• Maps a square into a quadrilateral • CANNOT map between 2 arbitrary quadrilaterals
úû
ùêë
é++++++
=úû
ùêë
éxybybxbbxyayaxaa
yxdyxd
y
x
3210
3210
),(),(
44
Bilinear Model
Bilinear model:• No chirping, but convergence of parallel lines
Other Polynomial Models
45
46
Motion Field Corresponding toDifferent 2-D Motion Models
Translation
Bilinear Projective
Affine
47
Region of Support for Motion Representation
Global:Entire motion field is represented by a few global parameters
Pixel-based:One MV at each pixel, with some smoothness constraint between adjacent MVs.
Region-based:Entire frame is divided into regions, each region corresponding to an object or sub-object with consistent motion, represented by a few parameters.
Block-based:Entire frame is divided into blocks, and motion in each block is characterized by a few parameters.
Practice Problem
Show that Eq. (1) can be simplified into the projective mapping Eq. (2) when the imaged object has a planar surface, that is
48
(1)
(2)
49