Motion in Video Presented By: Dr. S. K. Singh Indian Institute of Technology (B. H.U.) Varanasi Road Map Motion in Video Motion classification Motion in Video: Displacement, Velocity. Object Motion in Scene(3-D) Motion Model Image formation model TEMPORAL MOTION MODEL Calculation of displacement REGION OF SUPPORT MODELS OBSERVATIONAL MOTION MODEL Road Map Why Motion MODEL Motion Operation 2-D Motion and Apparent Motion Solution Of Aperture Problem 2-D MOTION ESTIMATION PROBLEM WITH 2-D MOTION 2-D MOTION ESTIMATION Representation Of Forward And Backward Motion Estimation Road Map Estimation of forward and backward Motion NON PARAMETRIC MODEL BLOCK MATCHING METHOD Pixel Based Motion Estimation Feature based methods (Indirect Method") Pixel Based Motion Estimation PEL-RECURSIVE METHOD Optical Flow Based Method For Motion Estimation Multiple Choice Question Motion in Video Motion is a change in position of object with respect to time and its reference point(like origin or other point ). It is described in terms of displacement (d) velocity(V), acceleration(a), and time(t). Motion is observed by attaching a frame of reference(F1) to a body and measuring its change in position relative to another reference frame(F2). Motion classification In physics, Motion in the universe is described through two sets of apparently (superficial) contradictory laws of mechanics. Motions of all large scale and familiar objects in the universe ( projectiles, planets, cells, and humans) are described by classical mechanics. Whereas the motion of very small atomic and sub-atomic objects is described by quantum mechanics. Motion in Video: Displacement, Velocity Object Motion in Scene MOTION Motion is a fundamental technique to understanding the scene(3-D Object Motion). It also helps to provides a sparse representation of scene with respect to time (t). A time varying image is such that the spatial intensity pattern changes with time. Hence time varying image is spatial temporal intensity pattern. It is denoted by S c (X 1, X 2, T), where X 1 and X 2 is spatial coordinates and t is time coordinate. Cont Source: macroevolution.livejournal.comSource: macroevolution.livejournal.com 501 Motion Model A motion model is used to describe object motion in an image sequence. Fundamental Motion Model 1.SPATIAL MOTION MODEL The main aim to estimate the motion of image points in the 2-D motion (apparent trajectory motion). Such motion is combination of projection s of the motion of objects in a 3-D scene and of 3-D camera motion. Cont Although camera motion affects the movement of all or almost all image points. The motion of 3-D object only affects a subset of image points compensated for by either estimating it or by physically measuring it at the camera. Image formation model Image formation model works on the following projection. Perspective Projection. Orthogonal projection. Motion model of 3-D object. Example: Rigid body with 3-d translation and rotation 3-D affine motion. TEMPORAL MOTION MODEL This model finds the trajectory(Path) of individual points drawn in the (x,y,t) space of an image sequence. It can be fairly arbitrary since they depends on object motion. The trajectory is linear function of velocity or displacement of moving object at each x. TEMPORAL MOTION MODEL: Calculation of displacement It can be represented by as follows: X(T)=x(t)+v t (t)*(T-t) =x(t)+d t,T (x) where d t,T (x)= v t (x)*(T-t). It is known as displacement vector(distance between moving pixel value). X(T)=x(t)+v t (t)*(T-t)+12 *a t (x).(T-t) 2. This model is based on two velocity (linear) variables and two acceleration (quadratic) variable. REGION OF SUPPORT MODELS The set of points (x, y,t)={(x 1, x 2, x 3, ..x n ),(y 1, y 2, y 3, y 4 y n ) and ( t 1, t 2, t 3, t 4 .t n )} are the values of spatial and temporal motion models. On the basis of these values,we find the region in given image by apply the region of support model of the moving object. Cont There are following types cases in the region 1.1-R=the whole image. 2.This model is suitable for estimation of camera induced motion in a simple static scene(image). 3.2-R= one pixel value. 4.3-R=Rectangular block of pixel. OBSERVATIONAL MOTION MODEL The observation model is based on the intensity variations of moving object with respect to time(t).During motion,object does not change their appearance. The value of intensity remains constant. We observe the continuous function of intensity along trajectory path. We find partial derivatives of intensity function (I) with respect to x, y, t and put zero. Note: Intensity function (I) must be remain constant through out trajectory path. It means I/s=0 Ix(v 1 ) +Iy( v 2 )+It=(x,y) T v +It =0. = (x,y) T = spatial gradient Where v=(v 1,v 2 ) T Note: It can apply at single position (x, y) under constrained (one equation and two known). Why Motion MODEL Why Motion model is required for Digital Video Processing? It is mathematical model describes various event in video processing system in parametric form. This is used to estimate these parameter( Velocity vector, displacement vector etc.) Motion Operation Motion model consist of following motion operation. Translation. Affine. Perspective. Rotation 2-D Motion and Apparent Motion 2-D Motion(Actual Motion) 2-D motion gives emphasize on moving object in (2-D) image plane. It is known as projected motion or pattern motion. There is no relative motion. Apparent Motion( Superficial Motion) Apparent motion emphasizes on motion of object in 2-D image plane projected from 3-D scene. It is not real motion of object what we observe is known as apparent(superficial motion ) into 2-D image plane. It deals with relative motion between moving object and observer (device like camera). Solution Of Aperture Problem The resolution of aperture problem given by a simple process known as intersection of constraints (IOC) for solving the aperture problem. IOC method uses only the local motions of two edge pieces to compute the global motion by finding the intersection of all possible global motions. It consistent with the two local motions detected. Cont The global motion velocity of the moving object can be simply calculated using IOC from the two edges of the 2D object associated with the top vertex of the object. As the outcome of aperture problem, the motion of a pictorial object composed of many components of different orientations. It known as pattern or global motion cannot be detected. It is shown below 2-D MOTION ESTIMATION PROBLEM WITH 2-D MOTION 2-D MOTION ESTIMATION BACKWARD ESTIMATION If the motion vector V is defined from time t to t- lt. It is known as backward estimation. Representation Of Forward And Backward Motion Estimation In the below figure,which explain the motion of trajectory of moving object in 2-D plane. Here L is trajectory find the apparent motion trajectory with the help forward and backward motion. Estimation of forward and backward Motion In below figure.1 the point A has same spatial coordinates all frames. Here L denotes the path (trajectory) of moving object in 3-D scene. There are two velocity vectors V f and V b known as forward and backward velocity vector. The compact notation of spatial coordinate S=(x, y). We will show the arrow from A which is same as S. A A A VbVb VfVf L Frame 1 Frame 3 Main frame=Frame 2 NON PARAMETRIC MODEL According to this model,Whenever a moving object under 3-D projection. It is not consider under rigidness of object. It provides the concept of smoothness of object edge and surface. NOTE: It can be classified under deterministic and stochastic model. 1.Optical Flow Filed Estimation. 2.Block Motion Estimation. 3.Pel- Recursive Algorithm. BLOCK MATCHING METHOD The block based motion estimation is preferred over pixel based method in particular implementations. It searches best matching block location which is in fixed size block. It depends on the frame translation of block. TYPE OF BLOCK SEARCH ALGORITHM Full search block motion estimation algorithm. 2-D logarithmic search algorithm. Three steps search algorithm. New three steps search algorithm. Pixel Based Motion Estimation The methods for finding motion vectors can be categorized into two parts. Pixel based methods (Direct Method"). Feature based methods (Indirect Method"). Pixel based methods (Direct Method") Block-matching algorithm. Source :www.sciencedirect.com Pixel Based Motion Estimation Phase correlation(frequency domain methods). Pel -Recursive motion estimation Netravali and Robbins developed a pel recursive spatio-temporal steepest descent gradient technique in which the displacement of a pel was predicted from previously transmitted information. Since then various algorithms have been proposed to improve the performance of pel recursive motion estimation (PRME) techniques. The most important contribution was the modification of the steepest-descent algorithm which was introduced by Walker and Rao. Pel -Recursive motion estimation where dx and dy correspond to the horizontal and vertical components of the motion displacement. Assuming that are known for each x, y, t and defini Pixel Based Motion Estimation Real-time Frame Rate Up-conversion for Video Games (SIGGRAPH 2010 Slides)and.intercon.ru Feature based methods (Indirect Method") Indirect methods use features, like Corner detection. Match corresponding features between frames. Methodology used : The main objective of the statistical function is: To Remove matches that do not correspond to the actual motion. With a statistical function applied over a local or global area. Statistical functions that have been successfully used include RANSAC. Pixel Based Motion Estimation Horn-schunck method It seeks a motion vector that satisfies the optical flow field estimation with minimum pixel to pixel variation among the flow vector. Multipoint neighborhood method: Assuming that every pixel in small block surrounding a pixel has same motion vector(MV). Pel recursive Method: In this method, motion vector(MV) is updated from those of its pixels. PEL-RECURSIVE METHOD It is predicator corrector type displacement estimator. The predicator can be taken as the value of the motion estimate the previous pixel location. Linear combination of motion estimation in neighborhood of current pixel. Optical Flow Based Method For Motion Estimation OPTICAL FLOW The optical flow is pixel level representation model for motion pattern. In each point in the image is assigned a motion vector. There are following method for estimation of optical flow. Bayesian method. Gibbs random field motion estimation. Optical flow equation. Second order derivatives of optical flow field. Digital Video Processing Textbooks Yao Wang, Jrn Ostermann, Ya-Qin Zhang Prentice Hall, 2002 A.M. Tekalp, Prentice-Hall, 1995 References 1.Tekalp,A.Murat Digital Video Processing. P. cm (Prentice Hall signal processing series),ISBN , AL Bovik Handbook of image and digital video processing", academic press, A Harcourt Science and Technology Company,ISBN , Yao Wang, Jrn Ostermann,and Ya-Qin Zhang Digital Video processing published in Prentice Hall, 2002. References 4.R. G. Gonzalez and R. E. Woods Digital Image Processing. Addison Wesley,2nd edition, http://en.wikipedia.org/wiki/Motion_%28physics%29.http://en.wikipedia.org/wiki/Motion_%28physics%29 6.Oge Marques Practical Image and Video Processing Using MATLAB. John Wiley & Sons, Technology & Engineering, H. Gharavi and H. Reza-Alikhani Pel-Recursive Motion Estimation Algorithm National Institute of Standards & Technology (NIST)