Upload
e8xu
View
96
Download
0
Embed Size (px)
Citation preview
Paper Note
Real-time rgb-d camera relocalization via randomized ferns for keyframe encoding
Organized by: Jacky Liu
jjkka132.slide (at) gmail.com
03 Feb 2017
National Chung Cheng UniversityRVL (Robot Vision Laboratory)
Paper Note, Jacky Liu, 03 Feb 2017
Why to read?
• To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking “white wall” scene.
• This paper is the relocalization module of ElasticFusion.
03 Feb 2017 2
[1] Whelan, Thomas, et al. "ElasticFusion: Dense SLAM Without A Pose Graph." Robotics: science and systems. Vol. 11. 2015.
Paper information
• MLA referenceGlocker, Ben, et al. "Real-time rgb-d camera relocalization via randomized ferns for keyframeencoding." IEEE transactions on visualization and computer graphics 21.5 (2015): 571-583.
• Citation until 03 Feb 2017: 9 times
• Key words1. Camera relocalization
2. Tracking recovery
3. Dense tracking and mapping
4. Marker-free augmented reality
03 Feb 2017 3
Outline
1. Real-time method
2. Compact representation of randomized fern
3. No spatial sampling heuristics
03 Feb 2017 4
Keyframe-based relocalization via randomized ferns for frame encoding
1. Frame encoding
2. Frame dissimilarity via Hamming Distance
3. Precision/Recall of BlockHD
4. Harvesting Keyframes
5. Tracking Recovery via pose retrieval
6. Fern Construction
03 Feb 2017 5
Thinking behind topics
A. Frame encoding→ Efficient way to represent frames, instead of using whole
image.
B. Frame dissimilarity via Hamming Distance→ How to compare two frames?
C. Precision/Recall of BlockHD→ BlockHD encoding property
D. Harvesting Keyframes→ How to determine “how densely we record the keyframe?”
E. Tracking Recovery via pose retrieval→ How to use encoded datastructure?
F. Fern Construction→ Detail of randomize fern method
03 Feb 2017 6
A - Frame encoding
03 Feb 2017 7
…m blocks (fern)
n node
Frame code: 𝑏𝑙𝑜𝑐𝑘1 𝑏𝑙𝑜𝑐𝑘2 …𝑏𝑙𝑜𝑐𝑘𝑚 = 0010 1101 … 0100
Frame id in order: 1 2 3 … k Code table for frame id lookup
B - Frame dissimilarity via Hamming Distance
03 Feb 2017 8
𝐵𝑙𝑜𝑐𝑘𝐻𝐷 𝑏𝐶𝐼 , 𝑏𝐶𝐽 =1
𝑚
𝑘=1
𝑚
𝑏𝐹𝑘𝐼 ≡ 𝑏𝐹𝑘
𝐽
Frame code: 𝑓𝑟𝑎𝑚𝑒𝐼 = 0010 1101 … 0100Frame code: 𝑓𝑟𝑎𝑚𝑒𝐽 = 0110 1001 … 0100
𝑏1 𝑏2 𝑏𝐶
Ex.Frame_I has 3 blocksHamming distance between frame I and J is 2
𝐵𝑙𝑜𝑐𝑘𝐻𝐷 𝑓𝑟𝑎𝑚𝑒𝐼, 𝑓𝑟𝑎𝑚𝑒𝐽 =1
𝐶
𝑘=1
𝐶
𝑓𝑟𝑎𝑚𝑒𝐼 ≡ 𝑓𝑟𝑎𝑚𝑒𝐽 =1
3∗ 2 = 0.66
C - Precision/Recall of BlockHD
code length ↗
change that two frame with low BlockHD ↘
precision ↗
recall ↘
vise versa
03 Feb 2017 9
Frame code: 𝑓𝑟𝑎𝑚𝑒𝐼 = 0010 1101 … 0100Frame code: 𝑓𝑟𝑎𝑚𝑒𝐽 = 0110 1001 … 0100
D - Harvesting Keyframes
• How to determine “how densely we record the keyframe?”
• Avoid to use spatial sampling heuristics=> non optimal scene coverage
• Minimum BlockHD K_I
𝐾𝐼 = min∀𝐽𝐵𝑙𝑜𝑐𝑘𝐻𝐷 𝑏𝐶
𝐼 , 𝑏𝐶𝐽= min∀𝐽(𝑚 − 𝑞𝐼𝐽𝑚)
03 Feb 2017 10
If 𝐾𝐼 > 𝑡, which t is predefined thresholdadd current frame as new keyframe
elsediscard current frame
E - Tracking Recovery via pose retrieval
• When tracking fails for an incoming frame, use most similar keyframe to retrieve pose proposals for tracking recovery.
• If reinitialization is unsuccessful for all proposed poses => discard current frame, repeat recovery until tracking is recovered
• Possible enhancementActively hint user to move the camera into areas of good keyframe coverage.
03 Feb 2017 11
F - Fern Construction
• Randomized parameters• Sampling pixel position x (uniform distribution)
• Number of ferns m
• Keyframe acceptance threshold t
• Number of neighbors k
03 Feb 2017 12
Experiment overview
• System based on KinectFusion (model based ICP)
• Two distinct evaluation set• Tracked frame for simulating harvesting of keyframes
• Performance evaluation
• Challenges of dataset1. Ambiguties (repeated steps in ‘Stairs’)
2. Specularities (reflections in ‘RedKitchen’)
3. Motion blur
4. Lighting conditions
5. Flat surfaces
6. Sensor noise
03 Feb 2017 13
Experiment – determine loss
• Plausibility check on the magnitude of camera motion and on the ICP residual error.
• If the relative motion or residual is too large, ICP reports tracking loss and relocalization is initiated
03 Feb 2017 14
Performance
• Percentage of frames for which the pose was successfully recovered.
• Definition of “successful recovery”Compare estimated pose with ground truth
• Translation error < 2cm
• Rotation error < 2 degrees
• Averaging three run to count “randomize” part of the algorithm
03 Feb 2017 15
Pose proposal strategies
• Using which keyframe pose to do ICP?• NN (nearest neighbor) – keyframe with smallest BlockHD
• kNN (k nearest neighbor) – Exam each k keyframe with smallest BlockHD, and pick the one with smallest ICP residual
• WAP (weighted average pose)
03 Feb 2017 16
Result
• Percentages correspond to the number of successfully recovered frames
03 Feb 2017 17
Offline v.s. Real-time
• Offline training methods outperform proposed real-time method. However, offline method is limited to revisited scene.
03 Feb 2017 18
Scalability to larger scenes
• Sufficient for representing large scenes of more than 30m^3 and thousands of frames.
03 Feb 2017 21
Marker-free augmented reality
• AR become more applicable when robustness of KinectFusion been improved by relocalizationalgorithm.
03 Feb 2017 22
Conclusion
• Solve the main cause of tracking failure in KinectFusion
• Compact frame encoding
• Lower performance compare to offline method
Future
• frame-to-frame tracking could give additional information about the relative pose trajectory.
• Loop closure
03 Feb 2017 23
Recall the goal of reading this paper
• To improve the robustness of ElasticFusion[1], one needs to study why relocalization module failed from recover after taking “white wall” scene.
• ElasticFusion loose track (https://youtu.be/mOmqd82EfDY)
• After meet the door, ICP report tracking loss.
• Relocalization module perform recovery.
• No loop closure to fix the tracking drift.
• To improve the robustness of ElasticFusion=> Apply loop closure. Define “uncertainty” of frame sequence, and use it to relocate path.
03 Feb 2017 24