Marker Detection and Tracking for Augmented Reality Applications!Oliver Toole, Dave Dolben!

Department of Computer Science, Stanford University

Motivation

Marker Tracking

Marker Detection!•  Explore the technology behind adding content

to a live video in real-time •  Detect and track “fiducial” markers in video

stream, estimate homography and replace with artificial imagery

•  Simple SIFT/SURF feature matching between marker image and video

•  Pre-compute marker keypoints/descriptors •  Match with descriptors on video frames •  Filter matches with distance ratio test and RANSAC •  Compute homography between marker image and

video frame (could estimate 3D with camera extrinsics)

•  KL Tracking of keypoints between frames:

•  Estimates displacement (ignore affine transformation) of small windows around keypoints by minimizing the error between two consecutive frames

•  Stop tracking keypoints when minimized error is too high

ε = [J(Ax − d)− I(x)]2wdxW∫

•  Let A=I, approximate I(x) with truncated Taylor series

•  Affine Consistency Check: Estimate displacement AND affine transform between first and current frame

•  Stop tracking keypoints when minimized error is too high

•  Can be computed iteratively by solving system of equations

Results/Comments!

I(x − d) = I(x)− g ⋅d

•  SIFT is slower than 30fps •  KL Tracking operates well on textured

windows and corners •  SIFT keypoints (blobs) not the best for

tracking, OpenCV’s goodFeaturesToTrack() is better BUT they are harder to match

•  Tracking is imperfect, even with OpenCV’s implementation of KL tracking. Maintaining correspondence for homography is hard

•  Other tracking methods, such as per-frame contour-based recognition might be better suited for binary fiducial markers

References!•  Bruce D. Lucas and Takeo Kanade. “An

Iterative Image Registration Technique with an Application to Stereo Vision.” International Joint Conference on Artificial Intelligence, pages 674-679, 1981.

•  Carlo Tomasi and Takeo Kanade. “Detection and Tracking of Point Features.” Carnegie Mellon University Technical Report CMU-CS-91-132, April 1991.

•  Jianbo Shi and Carlo Tomasi. “Good Features to Track.” IEEE Conference on Computer Vision and Pattern Recognition, pages 593-600, 1994.