Transcript
Page 1: Should we still do sparse-feature based SLAM?

Should we still do sparse-feature based SLAM?

Raúl Mur Artal PhD. student, University of Zaragoza

The Future of Real-Time SLAMICCV 2015 Workshop

Page 2: Should we still do sparse-feature based SLAM?

Feature-Based SLAM Direct SLAM

Minimize Feature Reprojection Error

Sparse Reconstruction

PTAM, ORB-SLAM

Minimize Photometric Error

Semi Dense / Dense Reconstruction

DTAM, LSD-SLAM, DPPTAM

Feature-based/Direct SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 3: Should we still do sparse-feature based SLAM?

Outline

1) Monocular ORB-SLAM (Mur-Artal, Montiel, Tardos T-RO 2015)

2) Monocular Semi Dense Mapping (Mur-Artal, Tardos, RSS 2015)

3) Stereo/RGB-D ORB-SLAM (unpublished)

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 4: Should we still do sparse-feature based SLAM?

Motivation

Parallel Tracking and Mapping (PTAM)G. Klein and D. Murray Parallel Tracking and Mapping for Small AR Workspaces ISMAR 2007.

First keyframe-based monocular SLAM

Bundle adjustment is possible in real-time

Relocalisation with little invariance to viewpoint

No loop closing/detection mechanism

Small scale operation

User intervention for map initialization and a dominant plane

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 5: Should we still do sparse-feature based SLAM?

Motivation

Bags of Binary Words (DBoW)D. Gálvez-López and J. D. TardósBags of Binary Words for Fast Place Recognition in Image SequencesT-RO 2012

Scale Drift-Aware Loop ClosingH. Strasdat, J. M. M. Montiel and A. J. DavisonScale Drift-Aware Large Scale Monocular SLAMRSS 2010

Covisibility GraphH. Strasdat, A. J. Davison, J. M. M. Montiel , K. KonoligeDouble Window Optimization for Constant Time Visual SLAMICCV 2011

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 6: Should we still do sparse-feature based SLAM?

ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 7: Should we still do sparse-feature based SLAM?

ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 8: Should we still do sparse-feature based SLAM?

ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 9: Should we still do sparse-feature based SLAM?

ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 10: Should we still do sparse-feature based SLAM?

ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 11: Should we still do sparse-feature based SLAM?

ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 12: Should we still do sparse-feature based SLAM?

ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 13: Should we still do sparse-feature based SLAM?

Automatic Map Initialization

OR

B-S

LA

MP

TAM

Model Selection

Homography(Planar, Low Parallax)

Fundamental Matrix(General)

https://youtu.be/UH8KXK8U_Ko

Page 14: Should we still do sparse-feature based SLAM?

Tracking I: Occlusion Handling

OR

B-S

LA

MP

TAM

Covisibility Graph

Track only a local map(potentially visible)

MapPointMean Viewing Direction

Do not project iffurther than 60º

https://youtu.be/UH8KXK8U_Ko?t=16s

Page 15: Should we still do sparse-feature based SLAM?

Tracking II: KeyFrame Insertion

OR

B-S

LA

MP

TAM

Survival of the FittestKeyFrame Selection

Little Restrictive Insertion(no distance threshold)

Restrictive Culling(redundant keyframes

detection)

https://youtu.be/UH8KXK8U_Ko?t=46s

Page 16: Should we still do sparse-feature based SLAM?

Relocalisation

OR

B-S

LA

MP

TAM

Bags of Binary Words

Same ORB thanTracking and Mapping

High Viewpoint Invariance

(ORB)

https://youtu.be/UH8KXK8U_Ko?t=1m11s

Page 17: Should we still do sparse-feature based SLAM?

ORB-SLAM

KITTI Dataset. Sequence 00

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

https://youtu.be/8DISRmsO2YQ

Page 18: Should we still do sparse-feature based SLAM?

RMS KeyFrame Position Error (cm)

Median over 5 executions

TUM RGB-D Benchmark

Tracking failure

Page 19: Should we still do sparse-feature based SLAM?

Comparison with RGBD-SLAM

F. Endres, J. Hess, J. Sturm, D. Cremers and W. Burgard3-D Mapping with an RGB-D CameraIEEE Transaction on Robotics, 2014.

Use depth information!

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 20: Should we still do sparse-feature based SLAM?

TUM RGB-D Benchmark

RMSE (cm)

RGB-D SLAM trajectoriesfrom the benchmark website

Page 21: Should we still do sparse-feature based SLAM?

Comparison with LSD-SLAM

State-of-the-art in Direct SLAM

J. Engel, T. Schöps, D. CremersLSD-SLAM: Large-Scale Direct Monocular SLAMEuropean Conference on Computer Vision (ECCV), 2014.

Use directly pixel intensities!

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 22: Should we still do sparse-feature based SLAM?

TUM RGB-D Benchmark

RMSE (cm)

Page 23: Should we still do sparse-feature based SLAM?

Comparison with LSD-SLAM

OR

B-S

LA

ML

SD

-SL

AM

fr3/structure_texture_near

https://youtu.be/UH8KXK8U_Ko?t=1m40s

Page 24: Should we still do sparse-feature based SLAM?

Why should we still use features?

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Robustness

Reliable two-view monocular initialization

Good invariance to viewpoint and illumination

Less affected by auto-gain and auto-exposure

Less affected by dynamic elements

Accuracy

Bundle adjustment (joint map-trajectory optimization)

Place Recogniton (loop detection, relocalization)

Bags of Words

But sparse reconstructions ...

Page 25: Should we still do sparse-feature based SLAM?

Probabilistic Semi-Dense Mapping

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 26: Should we still do sparse-feature based SLAM?

Probabilistic Semi-Dense Mapping

KeyFrame High Gradient Pixels Inverse Depth Map& uncertainty

?

Compute each inverse depth map from scratch using neighbor keyframes

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 27: Should we still do sparse-feature based SLAM?

Probabilistic Semi-Dense Mapping

Epipolar search in neighbor keyframes

IntensityCompare Gradient Modulo

Gradient Direction

Generate an inverse depth hypothesis per neighbor keyframe

Image noiseUncertainty Parallax

Matching ambiguity

Per Pixel Operations

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 28: Should we still do sparse-feature based SLAM?

Probabilistic Semi-Dense Mapping

Per Pixel Operations

Consistent Hypotheses Fusion

outlier

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 29: Should we still do sparse-feature based SLAM?

Probabilistic Semi-Dense Mapping

Per Pixel Operations

Intra-Keyframe

Neighbor KF

Neighbor KF

Neighbor KF

Inter-Keyframe

Smoothing

Outliers Detection

Inverse Depth Map Inverse Depth Map

Inverse Depth Map

Inverse Depth Map

Inverse Depth Map

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 30: Should we still do sparse-feature based SLAM?

Probabilistic Semi-Dense Mapping

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

https://youtu.be/HlBmq70LKrQ

Page 31: Should we still do sparse-feature based SLAM?

Stereo/RGBD ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

LeftImage

Right Image

ExtractORB

ExtractORB

StereoMatching

matchstereo keypoints: (u

L,v

L,u

R)

no match monocular keypoints: (uL,v

L)

Stereo (rectified)

Image

Depth-map

ExtractORB

GenerateStereo

Coordinate

RGBD

Bundle Adjustmentwith monocular and

stereo measurements

validdepth stereo keypoints: (u

L,v

L,u

R)

invaliddepth

monocular keypoints: (uL,v

L)

Page 32: Should we still do sparse-feature based SLAM?

Stereo/RGBD ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Close stereo keypoints (depth < threshold)Triangulation: 1 Frame

Fix scale Pure rotations

Agility

Far stereo keypoints (depth > threshold)Triangulation: 2 Keyframes

Moderate scale fix

Improves rotation estimation

Monocular keypoints (no stereo information)Triangulation: 2 Keyframes

Robustness to sensor limitations (RGBD)

Robustness to matching failure (stereo)

Page 33: Should we still do sparse-feature based SLAM?

Stereo/RGB-D ORB-SLAM

Monocular Stereo

Scale drift!EstimationGround Truth

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 34: Should we still do sparse-feature based SLAM?

Stereo ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

https://youtu.be/51NQvg5n-FE

Page 35: Should we still do sparse-feature based SLAM?

Stereo/RGB-D ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 36: Should we still do sparse-feature based SLAM?

Stereo ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

https://youtu.be/dF7_I2Lin54

Page 37: Should we still do sparse-feature based SLAM?

RGBD ORB-SLAM

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

https://youtu.be/LnbAI-o7YHk

Page 38: Should we still do sparse-feature based SLAM?

Publications & Software

Monocular ORB-SLAM R. Mur-Artal, J. M. M. Montiel and J. D. Tardos. A versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics. 2015

Open-source github.com/raulmur/ORB_SLAM

Monocular Semi-Dense Mapping R. Mur-Artal and J. D. Tardos. Probabilistic Semi-Dense Mapping from Highly Accurate Feature-Based Monocular SLAM. Robotics: Science and Systems. 2015

Stereo/RGBD ORB-SLAM Unpublished

Open-source coming soon!

The Future of Real-Time SLAM (ICCV'15 Workshop). Raúl Mur Artal. University of Zaragoza.

Page 39: Should we still do sparse-feature based SLAM?

Should we still do sparse-feature based SLAM?

Raúl Mur Artal PhD. student, University of Zaragoza

The Future of Real-Time SLAMICCV 2015 Workshop


Recommended