tracking
TP15 - TrackingComputer Vision, FCUP, 2013Miguel CoimbraSlides
by Prof. Kristen Grauman
OutlineToday: TrackingTracking as inferenceLinear models of
dynamicsKalman filtersGeneral challenges in trackingCS 376 Lecture
26 TrackingTracking: some applications
Body pose tracking, activity recognitionSurveillanceVideo-based
interfacesMedical apps
Censusing a bat population
Kristen GraumanWhy is tracking challenging? Optical flow for
tracking?If we have more than just a pair of frames, we could
compute flow from one to the next:But flow only reliable for small
motions, and we may have occlusions, textureless regions that yield
bad estimates anyway
CS 376 Lecture 26 TrackingMotion estimation techniquesDirect
methodsDirectly recover image motion at each pixel from
spatio-temporal image brightness variationsDense motion fields, but
sensitive to appearance variationsSuitable for video and when image
motion is small
Feature-based methodsExtract visual features (corners, textured
areas) and track them over multiple framesSparse motion fields, but
more robust trackingSuitable when image motion is large (10s of
pixels)
6CS 376 Lecture 26 TrackingFeature-based matching for motion
Interesting pointBest matching neighborhood
Time tTime t+1Search windowSearch window is centered at the
point where we last saw the feature, in image I1.
Best match = position where we have the highest normalized
cross-correlation value.Kristen GraumanExample: A Camera MouseVideo
interface: use feature tracking as mouse replacement
User clicks on the feature to be tracked Take the 15x15 pixel
square of the feature In the next image do a search to find the
15x15 region with the highest correlation Move the mouse pointer
accordingly Repeat in the background every 1/30th of a second
James Gips and Margrit
Betkehttp://www.bc.edu/schools/csom/eagleeyes/Kristen
GraumanExample: A Camera MouseSpecialized software for
communication, games
James Gips and Margrit
Betkehttp://www.bc.edu/schools/csom/eagleeyes/Kristen GraumanA
Camera MouseSpecialized software for communication, gamesJames Gips
and Margrit Betkehttp://www.bc.edu/schools/csom/eagleeyes/
Kristen GraumanFeature-based matching for motionFor a discrete
matching search, what are the tradeoffs of the chosen search window
size?
Which patches to track?Select interest points e.g. cornersWhere
should the search window be placed?Near match at previous frameMore
generally, taking into account the expected dynamics of the
object
Kristen GraumanCS 376 Lecture 26 TrackingDetection vs.
tracking
t=1t=2t=20t=21Kristen Grauman12CS 376 Lecture 26
TrackingDetection vs. tracking
Detection: We detect the object independently in each frame and
can record its position over time, e.g., based on blobs centroid or
detection window coordinatesKristen Grauman13CS 376 Lecture 26
TrackingDetection vs. tracking
Tracking with dynamics: We use image measurements to estimate
position of object, but also incorporate position predicted by
dynamics, i.e., our expectation of objects motion pattern.Kristen
Grauman14CS 376 Lecture 26 TrackingDetection vs. tracking
Tracking with dynamics: We use image measurements to estimate
position of object, but also incorporate position predicted by
dynamics, i.e., our expectation of objects motion pattern.Kristen
Grauman15CS 376 Lecture 26 TrackingTracking with dynamicsUse model
of expected motion to predict where objects will occur in next
frame, even before seeing the image.Intent: Do less work looking
for the object, restrict the search.Get improved estimates since
measurement noise is tempered by smoothness, dynamics
priors.Assumption: continuous motion patterns:Camera is not moving
instantly to new viewpointObjects do not disappear and reappear in
different places in the sceneGradual change in pose between camera
and scene
Kristen GraumanCS 376 Lecture 26 TrackingTracking as
inferenceThe hidden state consists of the true parameters we care
about, denoted X.
The measurement is our noisy observation that results from the
underlying state, denoted Y.
At each time step, state changes (from Xt-1 to Xt ) and we get a
new observation Yt.
Kristen GraumanState vs. observationHidden state : parameters of
interestMeasurement : what we get to directly observe
Kristen GraumanCS 376 Lecture 26 TrackingTracking as
inferenceThe hidden state consists of the true parameters we care
about, denoted X.
The measurement is our noisy observation that results from the
underlying state, denoted Y.
At each time step, state changes (from Xt-1 to Xt ) and we get a
new observation Yt.
Our goal: recover most likely state Xt givenAll observations
seen so far.Knowledge about dynamics of state transitions.
Kristen Grauman
Time tTime t+1Tracking as inference:
intuitionBeliefMeasurementCorrected predictionKristen Grauman
old beliefmeasurementBelief: predictionCorrected
predictionBelief: predictionmeasurementTracking as inference:
intuition
Time tTime t+1Kristen GraumanCS 376 Lecture 26
TrackingIndependence assumptionsOnly immediate past state
influences current state
Measurement at time t depends on current statedynamics
modelobservation model
Kristen GraumanCS 376 Lecture 26 TrackingPrediction:Given the
measurements we have seen up to this point, what state should we
predict?
Correction:Now given the current measurement, what state should
we predict?
Tracking as inference
Kristen GraumanCS 376 Lecture 26 TrackingQuestionsHow to
represent the known dynamics that govern the changes in the
states?
How to represent relationship between state and measurements,
plus our uncertainty in the measurements?
How to compute each cycle of updates?Representation: Well
consider the class of linear dynamic models, with associated
Gaussian pdfs.
Updates: via the Kalman filter.Kristen GraumanNotation
reminderRandom variable with Gaussian probability distribution that
has the mean vector and covariance matrix .x and are d-dimensional,
is d x d.
d=2
d=1If x is 1-d, we just have one parameter - the variance:
2Kristen GraumanCS 376 Lecture 26 TrackingLinear dynamic
modelDescribe the a priori knowledge about System dynamics model:
represents evolution of state over time.
Measurement model: at every time step we get a noisy measurement
of the state.
n x nn x 1n x 1m x nn x 1m x 1Kristen GraumanCS 376 Lecture 26
TrackingExample: randomly drifting pointsConsider a stationary
object, with state as positionPosition is constant, only motion due
to random noise term.State evolution is described by identity
matrix D=I
CS 376 Lecture 26 TrackingExample: Constant velocity (1D
points)
timemeasurementsstates1 d position 1 d position Kristen
GraumanState vector: position p and velocity v
Measurement is position onlyExample: Constant velocity (1D
points)
Kristen Grauman29CS 376 Lecture 26 TrackingQuestionsHow to
represent the known dynamics that govern the changes in the
states?
How to represent relationship between state and measurements,
plus our uncertainty in the measurements?
How to compute each cycle of updates?Representation: Well
consider the class of linear dynamic models, with associated
Gaussian pdfs.
Updates: via the Kalman filter.Kristen GraumanThe Kalman
filterMethod for tracking linear dynamical models in Gaussian
noiseThe predicted/corrected state distributions are GaussianOnly
need to maintain the mean and covarianceThe calculations are
easyKristen Grauman31CS 376 Lecture 26 TrackingKalman filterKnow
prediction of state, and next measurement Update distribution over
current state.Know corrected state from previous time step, and all
measurements up to the current one Predict distribution over next
state.Time advances: t++Time update(Predict)Measurement
update(Correct)Receive measurement
Mean and std. dev.of predicted state:
Mean and std. dev.of corrected state:Kristen GraumanCS 376
Lecture 26 Tracking1D Kalman filter: PredictionHave linear dynamic
model defining predicted state evolution, with noise
Want to estimate predicted distribution for next state
Update the mean:
Update the variance:
Lana Lazebnik33CS 376 Lecture 26 Tracking1D Kalman filter:
CorrectionHave linear model defining the mapping of state to
measurements:
Want to estimate corrected distribution given latest meas.:
Update the mean:
Update the variance:
Lana Lazebnik34CS 376 Lecture 26 TrackingPrediction vs.
correctionWhat if there is no prediction uncertainty
What if there is no measurement uncertainty
The measurement is ignored!The prediction is ignored!Lana
Lazebnik35CS 376 Lecture 26 Tracking
Kalman filter processingtimeo statex measurement* predicted mean
estimate+ corrected mean estimatebars: variance estimates before
and after measurementsConstant velocity modelposition
Time tTime t+1Kristen GraumanCS 376 Lecture 26 Tracking
Kalman filter processingtimeo statex measurement* predicted mean
estimate+ corrected mean estimatebars: variance estimates before
and after measurementsConstant velocity modelposition
Time tTime t+1Kristen GraumanCS 376 Lecture 26 Tracking
Kalman filter processingtimeo statex measurement* predicted mean
estimate+ corrected mean estimatebars: variance estimates before
and after measurementsConstant velocity modelposition
Time tTime t+1Kristen GraumanCS 376 Lecture 26 Tracking
Kalman filter processingtimeo statex measurement* predicted mean
estimate+ corrected mean estimatebars: variance estimates before
and after measurementsConstant velocity modelposition
Time tTime t+1Kristen GraumanCS 376 Lecture 26
Trackinghttp://www.cs.bu.edu/~betke/research/bats/
Kristen GraumanCS 376 Lecture 26 TrackingA bat census
http://www.cs.bu.edu/~betke/research/bats/Kristen Grauman41CS
376 Lecture 26 TrackingVideo
synopsishttp://www.vision.huji.ac.il/video-synopsis/
Kristen Graumanhttp://www.vision.huji.ac.il/video-synopsis/
42CS 376 Lecture 26 TrackingTracking: issuesInitializationOften
done manuallyBackground subtraction, detection can also be usedData
association, multiple tracked objectsOcclusions, clutter43CS 376
Lecture 26 TrackingTracking: issuesInitializationOften done
manuallyBackground subtraction, detection can also be usedData
association, multiple tracked objectsOcclusions, clutterWhich
measurements go with which tracks?
44CS 376 Lecture 26 TrackingTracking: issuesInitializationOften
done manuallyBackground subtraction, detection can also be usedData
association, multiple tracked objectsOcclusions, clutterDeformable
and articulated objects45CS 376 Lecture 26 TrackingRecall: tracking
via deformable contoursUse final contour/model extracted at frame t
as an initial solution for frame t+1Evolve initial contour to fit
exact object boundary at frame t+1Repeat, initializing with most
recent frame.
Visual Dynamics Group, Dept. Engineering Science, University of
Oxford.CS 376 Lecture 26 TrackingTracking:
issuesInitializationOften done manuallyBackground subtraction,
detection can also be usedData association, multiple tracked
objectsOcclusions, clutterDeformable and articulated
objectsConstructing accurate models of dynamicsE.g., Fitting
parameters for a linear dynamics modelDriftAccumulation of errors
over time47CS 376 Lecture 26 TrackingDrift
D. Ramanan, D. Forsyth, and A. Zisserman.Tracking People by
Learning their Appearance. PAMI 2007.48CS 376 Lecture 26
TrackingSummaryTracking as inferenceGoal: estimate posterior of
object position given measurementLinear models of dynamicsRepresent
state evolution and measurement modelsKalman filtersRecursive
prediction/correction updates to refine measurementGeneral tracking
challenges
