In my last article, we looked at feature-based visual SLAM (or indirect visual SLAM), which utilizes a series of keyframes and feature points to construct the world around the sensor. This approach initially enabled visual SLAM to run in real-time on consumer-grade computers and mobile devices, but with the increasing processing power of CPUs and the performance of cameras with lower noise, the desire for a denser point cloud representation of the world began to materialize through Direct Photogrammetric SLAM (or Direct SLAM). A denser point cloud would enable higher-accuracy 3D reconstruction of the world, stronger tracking especially in featureless environments, and scene changes (from weather and lighting). In this article, we will specifically look at the evolution of direct SLAM methods over the past decade, and some interesting trends emerging from it.
How Visual Slam Works
Direct SLAM originated from the idea of using all pixels from one camera frame to the next to reconstruct the world around the sensor, relying on photogrammetry principles. Instead of extracting feature points from images and tracking those feature points in 3D space, direct methods observe certain aspects of pixels (color, brightness, intensity gradients), and track the movement of those pixels from frame to frame. This approach transforms the problem being solved from minimizing geometric reprojection error, as in the case of indirect SLAM, to minimizing photometric error.
The direct visual SLAM solution we will review comes from a monocular (single camera) perspective. A stereo camera system would simplify some of the calculations required to obtain depth while providing accurate scaling on the map without extensive calibration.
For brevity, I have included some explanations of the basic concepts involved in visual SLAM:
- Structure from motion (SfM)
- Motion estimation
- Optical flow
Although these ideas help in a deeper understanding of some of the mechanisms, we will save them for another day.
The DTAM approach is one of the first real-time direct visual SLAM implementations, but it relies heavily on GPUs to make it work. Simply put, DTAM starts by taking several stereo baselines for each pixel until the first keyframe is obtained and an initial depth map with stereo measurements is created. Using this initial map, camera movement between frames is tracked by comparing images with the model view generated from the map. In each subsequent image frame, depth information is estimated for each pixel and optimized by minimizing the total depth energy.
The result is a model with depth information for each pixel, as well as an estimate of the camera pose. Because it tracks every pixel, DTAM produces a much denser depth map, appears much more robust in featureless environments, and is better suited for handling variations in focus and motion blur.
The following clip compares DTAM with Parallel Tracking and Mapping: PTAM, a classic feature-based visual SLAM method.
With fast motion, you can see tracking deteriorate as virtual objects placed in the scene jump around while the tracked feature points try to follow the scene changes (right panel). On the other hand, DTAM remains stable throughout the sequence because it tracks the entire scene, not just the detected feature points.
Since SLAM does not directly depend on sharp feature detection, when the scene’s focus changes, the tracked features disappear and tracking fails. This can occur in systems with cameras that have variable/automatic focus, and when images become blurry due to movement.
In this case, you can see the benefit of having a denser map, where accurate 3D reconstruction of the scene becomes possible.
Visual odometry and SLAM
We will start seeing more references to visual odometry (VO) as it develops, and I want everyone to have the same understanding of the terminology. As explained in the previous article, visual SLAM is the process of localization (understanding the current location and pose) and mapping the environment simultaneously, using visual sensors. An important technique introduced by indirect visual SLAM (more specifically by Parallel Tracking and Mapping – PTAM) is to parallelize the tracking, mapping, and optimization tasks into separate threads, where one thread tracks while the other builds and optimizes the map. For the purposes of this discussion, VO can be considered to focus on the localization part of SLAM. The VO process will provide input that the machine uses to build the map. However, this process will require additional functions for map consistency and optimization.
Large-Scale Direct SLAM
Based on previous work on the use of semi-dense depth maps for visual odometry, Jakob Engel (et al.) proposed the idea of Large-Scale Direct SLAM. Instead of using all available pixels, LSD-SLAM looks at high-gradient areas of the scene (especially edges) and analyzes the pixels in those areas. The idea is that there is very little to track between frames in areas with low or uniform pixel gradients to estimate depth. For a single camera, the algorithm uses pixels from the main frame as the basis for stereo depth calculations.
The following image highlights areas with high intensity gradients, which appear as lines or edges, unlike indirect SLAM, which typically detects corners and clusters as features.