Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

PCL :: FeaturesMichael Dixon and Radu B. Rusu

July 1, 2011

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Outline

1. Introduction

2. Feature Estimation

3. Keypoint Detection

4. Combining Keypoints and Features

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Introduction

In this session, we’ll talk about features and keypoints in PCL.

I Local feature estimationI Surface normal estimationI Point Feature Histogram descriptors

I Keypoint detectionI 3D SIFT keypoint detection

I Global feature estimationI Viewpoint Feature Histogram descriptors

Download the code and data at

http://dev.pointclouds.org/attachments/download/429/IROS-PCL-tutorial-code.zip

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Outline

1. Introduction

2. Feature Estimation

3. Keypoint Detection

4. Combining Keypoints and Features

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Local FeaturesWhat is a feature?

What is a feature?

In vision/perception, the word “feature” can mean manydifferent things. In PCL, “feature estimation” means:

I computing a feature vector based on each points localneighborhood

I or sometimes, computing a single feature vector for thewhole cloud

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Local FeaturesWhat is a feature?

Feature vectors can be anything from simple surface normals tothe complex feature descriptors need for registration or objectdetection.

Today, we’ll look at a couple of examples:

I Surface normal estimation (NormalEstimation)I Point Feature Histogram estimation (FPFHEstimation)I Viewpoint Feature Histogram estimation

(VFHEstimation)

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Local Features (1-2/5)Surface Normal Estimation. Theoretical Aspects: Basic Ingredients

I Given a point cloud with x,y,z 3D point coordinates

I Select each point’s k -nearest neighbors, fit a local plane,and compute the plane normal

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Local Features (1-2/5)Surface Normal Estimation. Theoretical Aspects: Basic Ingredients

I Given a point cloud with x,y,z 3D point coordinates

I Select each point’s k -nearest neighbors, fit a local plane,and compute the plane normal

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Local Features (3/5)Surface Normal and Curvature Estimation

Cj =∑k

i=1 ξi · (pi − pj)T · (pi − pj), p = 1

k ·∑k

i=1 pi

ξi =

e− d2

iµ2 , pi outlier

1, pi inlier

σp = λ0λ0+λ1+λ2

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Feature estimation basics

I Create the feature estimation objectI (Templated on the input and output point types)

I Set the search method to use for the underlyingneighborhood searches

I Create a k-D Tree and pass it to the feature estimatorI Specify the size of the local neighborhood to use when

computing the featuresI setRadiusSearch (r) or setNearestKSearch (k)

I Set the input pointsI Takes a shared pointer to a pcl::PointCloud

I ComputeI Outputs into a reference to a pcl::PointCloud

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

NormalEstimation example

Estimating surface normals using pcl::NormaLEstimationhttp://docs.pointclouds.org/1.1.0/classpcl_1_1_normal_estimation.html

SurfaceNormalsPtrestimateSurfaceNormals (const PointCloudPtr & input, float radius){

pcl::NormalEstimation<PointT, NormalT> normal_estimation;normal_estimation.setSearchMethod(pcl::KdTreeFLANN<PointT>::Ptr (new pcl::KdTreeFLANN<PointT>));

normal_estimation.setRadiusSearch (radius);

normal_estimation.setInputCloud (input);

SurfaceNormalsPtr normals (new SurfaceNormals);normal_estimation.compute (*normals);

return (normals);}

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

NormalEstimation results

$ ./test_feature_estimation robot.pcd -n 0.03

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Now let’s look at a more complex feature, like...

I Point Feature Histograms (PFH)

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Point Feature Histograms (PFH)A 3D feature descriptor

I For every point pair〈(ps,ns); (pt ,nt)〉, letu = ns, v = (pt−ps)×u, w = u×v

f0 = 〈v ,nj〉f1 = 〈u, pj − pi〉/||pj − pi ||f2 = ||pj − pi ||f3 = atan(〈w ,nj〉, 〈u,nj〉)

ihist =∑x≤3

x=0

⌊fx ·d

fxmax−fxmin

⌋· dx

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Point Feature Histograms (PFH)A 3D feature descriptor

f0 = 〈v ,nj〉f1 = 〈u, pj − pi〉/||pj − pi ||f2 = ||pj − pi ||f3 = atan(〈w ,nj〉, 〈u,nj〉)

ihist =∑x≤3

x=0

⌊fx ·d

fxmax−fxmin

⌋· dx

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Point Feature Histograms (PFH)A 3D feature descriptor

I For every point pair〈(ps,ns); (pt ,nt)〉, letu = ns, v = (pt−ps)×u, w = u×v

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Point Feature Histograms (PFH)Points lying on different geometric primitives

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

As an example, we’ll compute Fast Point Feature Histgramdescriptors (a faster approximation of PFH)

I Even though FPFH feature descriptors are more complex thansurface normals, the code is very similar

I The main difference: we need to set input normals

I First use pcl::EstimateNormalsI Then pass in a shared pointer to the resulting normals

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

FPHFEstimation example

LocalDescriptorsPtrcomputeLocalDescriptors (const PointCloudPtr & points, const SurfaceNormalsPtr & normals,

float feature_radius){

pcl::FPFHEstimation<PointT, NormalT, LocalDescriptorT> fpfh_estimation;fpfh_estimation.setSearchMethod(pcl::KdTreeFLANN<PointT>::Ptr (new pcl::KdTreeFLANN<PointT>));

fpfh_estimation.setRadiusSearch (feature_radius);fpfh_estimation.setInputCloud (points);

fpfh_estimation.setInputNormals (normals);

LocalDescriptorsPtr local_descriptors (new LocalDescriptors);fpfh_estimation.compute (*local_descriptors);

return (local_descriptors);}

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Outline

1. Introduction

2. Feature Estimation

3. Keypoint Detection

4. Combining Keypoints and Features

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

KeypointsWhat is a keypoint?

What is a keypoint?

I A keypoint (also known as an “interest point”) is simply apoint that has been identified as a relevant in some way.

I Whether any given point is considered a keypoint or notdepends on the keypoint detector in question.

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

KeypointsWhat is a keypoint?

There’s no strict definition for what constitutes a keypointdetector, but a good keypoint detector will find points whichhave the following properties:

I Sparseness: Typically, only a small subset of the points inthe scene are keypoints

I Repeatiblity: If a point was determined to be a keypoint inone point cloud, a keypoint should also be found at thecorresponding location in a similar point cloud. (Suchpoints are often called "stable".)

I Distinctiveness: The area surrounding each keypointshould have a unique shape or appearance that can becaptured by some feature descriptor

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

KeypointsWhy find keypoints?

What are the benefits of keypoints?

I Some features are expensive to compute, and it would beprohibitive to compute them at every point. Keypointsidentify a small number of locations where computingfeature descriptors is likely to be most effective.

I When searching for corresponding points, featurescomputed at non-descriptive points will lead to ambiguousfeature corespondences. By ignoring non-keypoints, onecan reduce error when matching points.

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

KeypointsKeypoint detection in PCL

There are a couple of keypoint detectors in PCL so far.I In this tutorial, we’ll focus on pcl::SIFTKeypoint

I A 3D adaptation of David Lowe’s SIFT keypoint detector

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

SIFT keypoint detection

The SIFT keypoint detector finds local extrema in adifference-of-Gaussians (DoG) scale-space. This scale spaceand its extrema are computed based on several parameters:

I the size of the smallest scaleI the number of times the scale doublesI the number of scales in between each doublingI the minimum absolute DoG value needed to qualify as a

keypoint

Now let’s look at an example of how to compute 3D SIFTkeypoints in PCL

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

SIFTKeypoints example

Computing keypoints on a cloud of pointsPointCloudPtrdetectKeypoints (const PointCloudPtr & points, const SurfaceNormalsPtr & normals,

float min_scale, int nr_octaves, int nr_scales_per_octave, float min_contrast){

pcl::SIFTKeypoint<PointT, pcl::PointWithScale> sift_detect;sift_detect.setSearchMethod(pcl::KdTreeFLANN<PointT>::Ptr (new pcl::KdTreeFLANN<PointT>));

sift_detect.setScales (min_scale, nr_octaves, nr_scales_per_octave);sift_detect.setMinimumContrast (min_contrast);sift_detect.setInputCloud (points);

pcl::PointCloud<pcl::PointWithScale> keypoints_temp;sift_detect.compute (keypoints_temp);

PointCloudPtr keypoints (new PointCloud);pcl::copyPointCloud (keypoints_temp, *keypoints);

return (keypoints);}

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Outline

1. Introduction

2. Feature Estimation

3. Keypoint Detection

4. Combining Keypoints and Features

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Combining keypoints and features

Keypoints and feature descriptors go hand-in-hand.

Let’s look at how to compute features only at a given set ofkeypoints.

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Estimating features at a subset of points

I Create the feature estimation objectI Set the search method to use for the underlying

neighborhood searchesI Specify the size of the local neighborhood to use when

computing the featuresI Set the search surface

I These points are used when searching for the input points’neighbors

I Set the input pointsI These are the points for each feature is computed

I Compute

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Keypoints and features

Computing local feature descriptors for a set of keypointsLocalDescriptorsPtrcomputeLocalDescriptors (const PointCloudPtr & points, const SurfaceNormalsPtr & normals,

const PointCloudPtr & keypoints, float feature_radius){

pcl::FPFHEstimation<PointT, NormalT, LocalDescriptorT> fpfh_estimation;fpfh_estimation.setSearchMethod (pcl::KdTreeFLANN<PointT>::Ptr (new pcl::KdTreeFLANN<PointT>));fpfh_estimation.setRadiusSearch (feature_radius);fpfh_estimation.setSearchSurface (points);fpfh_estimation.setInputNormals (normals);fpfh_estimation.setInputCloud (keypoints);

LocalDescriptorsPtr local_descriptors (new LocalDescriptors);fpfh_estimation.compute (*local_descriptors);

return (local_descriptors);}

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Global feature descriptors

Computing global feature descriptors

A global feature descriptor isI A single descriptor computed over the entire cloud.I Used to compare clouds for object recognition later in this

tutorialAs an example, we’ll be using Viewpoint Feature Histograms

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

VFHEstimation

Computing a VFH feature descriptorGlobalDescriptorsPtrcomputeGlobalDescriptor (const PointCloudPtr & points, const SurfaceNormalsPtr & normals){

pcl::VFHEstimation<PointT, NormalT, GlobalDescriptorT> vfh_estimation;vfh_estimation.setSearchMethod (pcl::KdTreeFLANN<PointT>::Ptr (new pcl::KdTreeFLANN<PointT>));vfh_estimation.setInputCloud (points);vfh_estimation.setInputNormals (normals);

GlobalDescriptorsPtr global_descriptor (new GlobalDescriptors);vfh_estimation.compute (*global_descriptor);

return (global_descriptor);}

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Testing

Use test_feature_estimation to run your code on a PCD file.The options are:

I -n Estimate surface normalsI -k Detect keypointsI -l Compute local descriptors

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Questions?

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

example

The slides after this point are random ones we didn’t end upusing...

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

PFH (5/5)Complexity Analysis

Complexity is high: O(k2). Optimizations to the rescue!

Unordered OrderedPoint Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Fast PFH (1/4)Basic Concepts

Re-formulate: FPFH(p) = SPF (p) + 1k∑k

i=11ωk· SPF (pk )

Point Feature Histograms (PFH) Fast Point Feature Histograms (FPFH)

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Fast PFH (2/4)Theoretical formulation

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Fast PFH (3/4)Noise Analysis

Synthetically noiseless Synthetically noisified

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Fast PFH (4/4)Noise Analysis

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Features (1/4)pcl::NormalEstimation<T> p;

I p.setInputCloud (data);

p.SetRadiusSearch (0.01);

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Features (2/4)pcl::NormalEstimation<T> p;

I p.setInputCloud (data);

p.SetRadiusSearch (0.01);

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Features (3/4)pcl::BoundaryEstimation<T,N> p;

I p.setInputCloud (data);p.setInputNormals (normals);

p.SetRadiusSearch (0.01);

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Features (4/4)pcl::PrincipalCurvaturesEstimation<T,N> p;

I p.setInputCloud (data);p.setInputNormals (normals);

p.SetRadiusSearch (0.01);

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Feature PersistenceWhat scale to choose ?

bad scale (too small) good scale

Selecting the right scale (k -neighborhood) is problematic:

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Local Features (4-5/5)Consistent Normal Orientation

Before

I Extended Gaussian ImageI Orientation consistent for:

1. registration2. feature estimation3. surface representation

I normals on theGaussian sphere

I should be in thesame half-space

Point Cloud Library (PCL)

Introduction Feature Estimation Keypoint Detection Combining Keypoints and Features

Local Features (4-5/5)Consistent Normal Orientation

Before After

(vSolet ′slookatsomecode : iewpoint − pi) · npi ≥ 0

or:

propagate consistencythrough an EMST

Point Cloud Library (PCL)