1. MOTION FLOW SEGMENTATION & ANALYSIS Advanced Topics in
Video Surveillance Yusuf Ziya UZUN
2. MOTIVATION DARPA Visual Media Reasoning (VMR)
3. WHAT IS VIDEO It is simply Sequence Of Images, literally:
Group of Pictures(GOP) I-Frame : Intra-coded frame P-Frame :
Predicted frame B-Frame : Bi-directional predicted frame B-Frame
< P-Frame < I-Frame
4. WHAT IS MOTION Motion: displacement, direction, velocity,
acceleration, time and speed Motion Vector: Projection vector of
motion from 3d to 2d Motion field: 3d motions projected to 2d
images; dependency on depth Motion Flow: Ideally equals to Optical
Flow http://en.wikipedia.org/wiki/Motion_(physics)
http://en.wikipedia.org/wiki/Motion_estimation
5. WHAT IS OPTICAL FLOW Optical Flow: distribution of the
apparent velocities of objects in an image Brightness, color Zoom
out Zoom in Pan right to left
http://www.mathworks.com/discovery/optical-flow.html
6. HOW TO ESTIMATE FLOW Why estimate motions? Track object
behavior, alignment, stabilization. How to estimate pixel motion
from image H to image I? Color constancy Looks same Brightness
constancy Grayscale images Small motions Not to move far
https://courses.cs.washington.edu/courses/csep576/05wi/lectures/motion.pdf
7. SOME OF SEGMENTATION METHODS Background Subtraction Gaussian
Distribution (PDF) Shot Boundary Detection Find Good Key-Frames
(I-Frame) Feature Detection / Extraction Sobel Filter, SIFT Motion
Segmentation Clustering of Motion Sets Clustering K-means
Graph-based Segmentation Grouping, Cutting Superpixels Distance,
graph, clustering
8. PROBLEM The ground truth does not exist: the desired results
always depend on the user requirements and specifications. Even for
a fixed image, there may be more than one "best" segmentation
because the criteria defining the quality of a segmentation are
application dependent. -Pierre Soille
http://cvl.ice.cycu.edu.tw/meeting/2008.09.23.pdf
9. BACKGROUND SUBTRACTION Easy to implement, pretty fast,
simply filtering One threshold for image (constant for every frame,
no time dependency) What about Lighting changes, repetitive motions
from clutter and long-term scene changes? Adaptive background
mixture model http://en.wikipedia.org/wiki/Background_subtraction
http://www.cs.utexas.edu/~grauman/courses/fall2009/slides/lecture9_background.pdf
10. SHOT BOUNDARY DETECTION Hard cut, fade, dissolve How it
helps? Storyline, time based localization Searching show me all
films where there's a scene with a lion in it. Temporal
Segmentation FeatureType of Edit Hard Cuts Fades Dissolve Color
Histogram Differences X Edge Change Ratio X X X Standard Deviation
of Pixel Intensities X Contrast X
http://www.vis.uky.edu/~cheung/courses/ee639_fall04/readings/spie99.pdf
http://en.wikipedia.org/wiki/Shot_transition_detection
11. FEATURE DETECTION / EXTRACTION Interesting parts of image
Corners, edges, blobs Feature Candidates Scale Invariant Feature
Transform (SIFT) Speeded Up Robust Features (SURF) What is the
relation between Motion Segmentation and Feature Extraction? We
need to find good track points to create better segments!
12. MOTION SEGMENTATION Seperate moving objects from background
by using motion vectors(optical flow) Just split image N pieces.
Trajectory segmentation, Local, Global Horn and Schunck,
Kanade-Lucas-Tomasi(KLT) A local constraint to solve the aperture
problem. Aperture, Barber-pole (Motion vs Optical) Closer Objects
Have Bigger Velocity?
http://en.wikipedia.org/wiki/Barberpole_illusion
13. CLUSTERING Motion vector clustering for better segmentation
and tracking
14. GRAPH-BASED SEGMENTATION Affinity between pixels: Color
distance Weighted with gradients Take into account optical flow
From per pixel classifiers, etc. How to connect? Direct predecessor
Displaced along optical flow t - 1 t t + 1
http://www.videosegmentation.com/
15. APPLICATIONS Surveillance cameras Self driving cars
(Autonomous) Estimating 3D structures Recognizing events and
activities Facial expression recognition Video compression
16. EXAMPLE: AUTONOMOUS CAR Sebastian Thrun, Google Self
Driving Ca
17. REFERENCES Motion Segmentation: a Review, Luca ZAPPELLA
aXavier LLADa and Joaquim SALVI a Extracting representative motion
flows for effective video retrieval, Zhe Zhao Bin Cui Gao Cong Zi
Huang Heng Tao Shen The Computation of Optical Flow, S S Beauchemin
and J L Barron A Database and Evaluation Methodology for Optical
Flow, Simon Baker Daniel Scharstein J.P. Lewis Stefan Roth Michael
J. Black Richard Szeliski Online Motion Segmentation using Dynamic
Label Propagation Ali Elqursh, Ahmed Elgammal Performance of
Optical Flow Techniques JL Barron, DJ Fleet and SS Beauchemin
Background Subtraction Birgi Tamersoy Motion Estimation Motion and
optical flow Optical flow (motion vector) computation, Nilesh
Ghubade Optical Flow Estimation D.J. Fleet & A.D. Jepson, 2005
OPTICAL FLOW USING COLOR INFORMATION: PRELIMINARY RESULTS Kelson R.
T. Aires, Andre M. Santana, Adelardo A. D. Medeiros Effcient
Graph-Based Image Segmentation Pedro F. Felzenszwalb Graph-Based
Hierarchical Video Segmentation Matthias Grundmann, Vivek Kwatra,
Mei Han, Daniel Castro, Irfan Essa SLIC Superpixels* Radhakrishna
Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and
Sabine Susstrunk Comparison of Automatic Shot Boundary Detection
Algorithms Rainer Lienhart