1 Challenge the future
Basic Point Cloud Processing
Estimating Normal Vectors and Curvature Indicators
Ir. Pirouz Nourian
PhD candidate & Instructor, chair of Design Informatics, since 2010
MSc in Architecture 2009
BSc in Control Engineering 2005
Geo1004, Geomatics Master Track Directed by Dr. ir. Sisi Zlatannova
2 Challenge the future
How do we make a computer ‘see’ what we understand from this?
What do you make of this?
What is this all about?
3 Challenge the future
Data to Information: LIDAR to LOD2
How do we make sense out of such big data?
Image courtesy of GEOCONNEXION
?
4 Challenge the future
SOFTWARE APPLICATIONS
• FME, Safe Software
• LASTools, rapidlasso
• Point Cloud Library (PCL)
• Cloud Compare
• MeshLab
• LP360 QCoherent
A NEW VERSION OF TOIDAR! YES!
http://opentopo.sdsc.edu Information from:
5 Challenge the future
FME, Safe Software
transform, convert, translate, extract, integrate, automated, repeatable
6 Challenge the future
LASTools, rapidlasso
filtering, clipping, reprojecting, compression, classification, DSM, DTM, TIN, contours bare-earth
de facto standards for point cloud data:
.las .laz
7 Challenge the future
LP360, QCoherenet
LIDAR, Classification, Breaklines, Visualization, Extraction, Automatic
8 Challenge the future
Point Cloud Library PCL
point clouds, visualization, processing, segmentation, filtering, feature estimation, registration
Using this library in Rhino?
9 Challenge the future
Cloud Compare
Implements PCL and more methods, handy to use for point cloud processing
Image from software.informer.com
10 Challenge the future
MeshLab
Has some surface reconstruction methods, handy to have when working with point clouds.
Image from https://danieleferdani.wordpress.com/
12 Challenge the future
Part 1 finished
We will continue with one specific way of dealing with point
clouds and estimating normal vectors and curvatures…
14 Challenge the future
A Curvature-Based Approach to Point Cloud
Segmentation & Feature Detection
Removing
Outliers
Elevation
Classification
Slope
Classification
Aspect
Classification
Region-Growing
Segmentation
Point Cloud
of a Building
Noise
reduction
Forming
Neighbourhoods:
e.g. KNN or Range
Forming
Covariance
Matrices for
Neighbourhoods
Using PCA, Estimating:
Normal Vectors &
Curvature Indicators
Normal
Vectors
Curvature
Indicators
Eigen
System
Triangulation Mesh
Surface
15 Challenge the future
Different Approaches for Finding Fitting Planes/Edges
1. Using curvature values computed (estimated) by eigenvalues of covariance matrices, to run a segmentation algorithm based on region growing.
2. Using Octree Voxels, to detect edge voxels, remove them and create segments.
3. Using Hough transform on a 2.5D point cloud, converting to the parameter space and using DBScan clustering method to find clusters of parameters, each of which correspond to a segment in a point cloud.
4. And a few more in the literature…
16 Challenge the future
Different Approaches for Finding Fitting Planes/Edges
Region growing based on normals and curvature Segmentation using Octree voxels Hough transformation
17 Challenge the future
Different Approaches for Finding Fitting Planes/Edges:
our experience!
Curvature based region
growing
Voxel based region
growing
Hough transform
Noise management Fair Good Good
Density variation
management
Fair Fair Good
Efficiency Good Good Poor
Ambiguity of vantage
point management
Fair Good Good
Avoiding priori
knowledge
Good Good Poor
Ease of further
processing
Good Poor Poor
18 Challenge the future
Neighbourhoods of Points
Fixed Distance Neighbors (FDN) and K-Nearest Neighbors (KNN)
• Rabbani, T., van den Heuvel, F., & Vosselmann, G. (2006). Segmentation of point clouds using smoothness
constraint. International Archives of Photogrammetry, Remote Sensing and Spatial Information
Sciences, 36(5), 248-253.
• Pauling, Frederick, Michael Bosse, and Robert Zlot. "Automatic segmentation of 3d laser point clouds by
ellipsoidal region growing." Australasian Conference on Robotics and Automation. 2009.
The idea is to form neighborhoods based on [squared] distance These two options are usually provided. Apart from normal vector estimation KNN can be useful also for removal of outliers.
19 Challenge the future
Fitting Planes to Neighbourhoods
The idea is that the underlying surface is a 2-manifold; therefore it
resembles a 2D plane locally
Least Square Plane Fitting Estimation Problem
Principal Component Analysis Problem
Intuition: Defining plane as the locus of lines that have direction vectors perpendicular to a normal vector; considering an origin for the plane in questions, we can consider the plane as the locus of points A such that A-O is perpendicular to N.
20 Challenge the future
Fitting Planes to Neighbourhoods
The idea is that the underlying surface is a 2-manifold; therefore it
resembles a 2D plane locally, we look at ellipsoids showing local
dispersions
Images courtesy of Olga Sorkine
21 Challenge the future
How to estimate normals using PCA
The idea is that the underlying surface is a 2-manifold; therefore it
resembles a 2D plane locally
• Pauly, Mark, Markus Gross, and Leif P. Kobbelt. "Efficient simplification of point-sampled surfaces." Proceedings of the conference on Visualization'02. IEEE Computer Society, 2002.
• Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W. Surface reconstruction from unorganized points. SIGGRAPH 92, 1992
• Shaffer, E., Garland, M. Efficient Adaptive Simplification of Massive Meshes. IEEE Visualization 01, 2001
22 Challenge the future
How to estimate normals using PCA
We form a covariance matrix for each neighborhood, showing how
neighbors are dispersed around their average (centroid).
This will be a 3
by 3 symmetric
matrix!
23 Challenge the future
How to estimate normals using PCA
We form a covariance matrix for each neighborhood, showing how
neighbors are dispersed around their average (centroid).
This will be a 3
by 3 symmetric
matrix! Form an Eigen system for this matrix using a linear algebra library, and the first eigenvector corresponding to least eigenvalue will be the normal vector at each neighbourhood. Explanation follows…
PCL implementation: http://pointclouds.org/documentation/tutorials/normal_estimation.php
24 Challenge the future
How to estimate curvature using PCA
The idea is to use an indication of change along the normal vector
Jolliffe, I. Principle Component Analysis. Springer-Verlag, 1986
25 Challenge the future
How to do all this in code? • We try not to reinvent the wheel; the idea is to use free open
source libraries like Math.NET and Accord.NET
• We are not the first people dealing with such issues, these are
generally matters of data mining and machine learning
• We can find KNN neighbourhoods using Accord.NET http://accord-framework.net/
• We can find eigenvalues and eigenvectors using MetaNumerics.dll, MathNet.dll or Accord.NET
• You will receive example code implementing MetaNumerics
26 Challenge the future
Fitting Planes to Neighbourhoods
How do we minimize the error in fitting a plane to the
mentioned neighbourhood? First we define it…
Images courtesy of Olga Sorkine
27 Challenge the future
An explanation after Olga Sorkine:
• Input points:
• Centroid:
• Vectors from the centroid:
m
28 Challenge the future
Continued…
m m minimizes SSD and it can be shown that m is the centroid C
We can rewrite the problem as:
29 Challenge the future
Continued…
m
m minimizes SSD, formally it is the “arg min” of the following function (to be minimized), i.e. the argument that makes it reach its minimum. and it can be shown that m is the centroid C
Solving this turns out to be equal to
solving an Eigen system for the
covariance matrix; you can see how…
30 Challenge the future
Long story short…
We find eigenvalues and eigenvectors of the covariance matrices…
31 Challenge the future
Long story short…
We find eigenvalues and eigenvectors of the covariance matrices…
What was this all about? Segmenting the point cloud taking into account [underlying] surface variations, in search of smooth patches, made disjoint by edges (where we find high curvature/variation). Therefore we can use the above estimated vectors and values for such a segmentation.
34 Challenge the future
Continued from lecture notes of Olga Sorkine…
This S is our covariance
matrix, remember?
35 Challenge the future
•Constrained minimization – Lagrange multipliers
Continued from lecture notes of Olga Sorkine…
Given this equality in
differentiation of vector
valued functions
36 Challenge the future
•Constrained minimization – Lagrange multipliers
Continued from lecture notes of Olga Sorkine…
maximize f(x, y) subject to g(x, y) = c. Lagrangian: The prblem is transformed to finding a ‘’staionary point for the Langrangian at which the partial derivatives of Lagrangian function are zero.
37 Challenge the future
•Constrained minimization – Lagrange multipliers
Continued from lecture notes of Olga Sorkine…
This one means that the
normal vector is
normalized, meaning its
length is equal to 1; why,
because the dot product
of a vector by itself gives
the squared length
38 Challenge the future
How to form covariance matrices in code?
VB.NET (confirm and debug for yourself)
If(Pts.Count > 3) Then Dim CovMatrices As New list(Of Meta.Numerics.Matrices.SymmetricMatrix) For j As Integer=0 To Pts.Count - 1 Dim CovM As New Meta.Numerics.Matrices.SymmetricMatrix(3) Dim Neighbors As List(Of Integer) = DirectCast(Neigh(j), List(Of Integer)) Dim Centroid As New Point3d Dim NPts As New List(Of Point3d) For Each neighbor As Integer In Neighbors Centroid = Centroid + Pts(neighbor) NPts.Add(Pts(neighbor)) Next Centroid = Centroid / Neighbors.Count Dim CiCBar As New RectangularMatrix(3, Neighbors.Count) For k As Integer=0 To Neighbors.count - 1 Dim Diff As point3d = Pts(Neighbors(k)) - Centroid CiCBar(0, k) = Diff.X CiCBar(1, k) = Diff.y CiCBar(2, k) = Diff.z Next CovM = CiCBar.MultiplySelfByTranspose() CovM = (1 / (Neighbors.count - 1)) * CovM CovMatrices.Add(CovM) Next A = CovMatrices(0).ToArray() C = CovMatrices End If
39 Challenge the future
How to form covariance matrices in code?
C#.NET (confirm and debug for yourself)
if ((Pts.Count > 3)) { List<Meta.Numerics.Matrices.SymmetricMatrix> CovMatrices = new List<Meta.Numerics.Matrices.SymmetricMatrix>(); for (int j = 0; j <= Pts.Count - 1; j++) { Meta.Numerics.Matrices.SymmetricMatrix CovM = new Meta.Numerics.Matrices.SymmetricMatrix(3); List<int> Neighbors = (List<int>)Neigh(j); Point3d Centroid = new Point3d(); List<Point3d> NPts = new List<Point3d>(); foreach (int neighbor in Neighbors) { Centroid = Centroid + Pts(neighbor); NPts.Add(Pts(neighbor)); } Centroid = Centroid / Neighbors.Count; RectangularMatrix CiCBar = new RectangularMatrix(3, Neighbors.Count); for (int k = 0; k <= Neighbors.count - 1; k++) { point3d Diff = Pts(Neighbors(k)) - Centroid; CiCBar(0, k) = Diff.X; CiCBar(1, k) = Diff.y; CiCBar(2, k) = Diff.z; } CovM = CiCBar.MultiplySelfByTranspose(); CovM = (1 / (Neighbors.count - 1)) * CovM; CovMatrices.Add(CovM); } A = CovMatrices(0).ToArray(); C = CovMatrices; }
41 Challenge the future
Grid Surface Reconstruction
Pseudo-Code
Define a new Mesh with a list of Vertices and a list of Faces Mesh.Vertices=Points For j as Integer=0 to V-2//choose only bottom-left corners as pivot points For i As Integer=0 To U - 2 define n0,n1,n2,n3 As Integer n0 = j * U + I //bottom-left n1 = n0 + 1 //bottom-right n2 = n0 + U //top-right n3 = n1 + U //top-left Define face As new MeshFace(n0, n1, n3, n2) Mesh.Faces.AddFace(face) Next Next
42 Challenge the future
Grid Surface Reconstruction
VB.NET
Dim M As New Mesh M.Vertices.AddVertices(P) If Not C Is Nothing Then M.VertexColors.AppendColors(C.toArray) For j As Integer=0 To V - 2 For i As Integer=0 To U - 2 Dim n0,n1,n2,n3 As Integer n0 = j * U + i n1 = n0 + 1 n2 = n0 + U n3 = n1 + U Dim face As MeshFace = New MeshFace(n0, n1, n3, n2) M.Faces.AddFace(face) Next Next
with two nested loops
43 Challenge the future
Grid Surface Reconstruction
VB.NET
Dim M As New Mesh M.Vertices.AddVertices(P) If Not C Is Nothing Then M.VertexColors.AppendColors(C.toArray) For i As Integer=0 To U * (V - 1) - 1 If (i Mod u) < u - 1 Then Dim n0,n1,n2,n3 As Integer n0 = i n1 = n0 + 1 n2 = n0 + U n3 = n1 + U Dim face As MeshFace = New MeshFace(n0, n1, n3, n2) M.Faces.AddFace(face) End If Next
with one loop
44 Challenge the future
Grid Surface Reconstruction
C#.NET
Mesh M = new Mesh(); M.Vertices.AddVertices(P); if ((C != null)) M.VertexColors.AppendColors(C.toArray); for (int i = 0; i <= U * (V - 1) - 1; i++) { if ((i % u) < u - 1) { int n0 = 0; int n1 = 0; int n2 = 0; int n3 = 0; n0 = i; n1 = n0 + 1; n2 = n0 + U; n3 = n1 + U; MeshFace face = new MeshFace(n0, n1, n3, n2); M.Faces.AddFace(face); } }
with one loop
45 Challenge the future
Simple Estimation of Normal
Vectors Pseudo-Code
Form an empty list of normal vectors Define deviation as a double For each point as Point3d in the point cloud find neighbors fit a plane to neighbors Get the normal of this plane and put it out as the normal of the point form a vector from the vantage point VP to point=VP-point and call it dir if this normal.dir>0 then Add the normal to the list of normals Else Add –normal to the list of normals End Next
46 Challenge the future
Estimation Normal Vectors
C#.NET
List<Vector3d> Normals = new List<Vector3d>(); Point3dList PCList = new Point3dList(); PCList.AddRange(x); double Dev = MD; foreach (Point3d point in PCList) { dynamic Neighbors = PCList.FindAll(V => V.DistanceTo(point) < D); plane NP = default(Plane); Plane.FitPlaneToPoints(Neighbors, NP, Dev); if (NP.Normal * (VP - point) > 0) { Normals.Add(NP.Normal); } else { Normals.Add(-NP.Normal); } } A = Normals; B = PCList.FindAll(VT => VT.DistanceTo(x(654)) < D);
47 Challenge the future
Estimation Normal Vectors
VB.NET
Dim Normals As New list(Of Vector3d) Dim PCList As New point3dlist PClist.AddRange(x) Dim Dev As Double = MD For Each point As point3d In PClist Dim Neighbors = PClist.FindAll(Function(V) V.distanceto(point) < D) Dim NP As plane Plane.FitPlaneToPoints(Neighbors, NP, Dev) If NP.Normal * (VP - point) > 0 Then Normals.Add(NP.Normal) Else Normals.Add(-NP.Normal) End if Next A = Normals B = PClist.FindAll(Function(VT) VT.DistanceTo(x(654)) < D)
48 Challenge the future
Basics of Scripting in Rhino + GH
• VB.NET (Sub[routine], Function, ByVal, ByRef)
• C#.NET (Void, [Functions], [val],ref)
• Python (defs)
Basic concepts of systems and modules, Inputs & Outputs
Private Sub RunScript(ByVal x As Object, ByVal y As Object, ByRef A As Object) End Sub '<Custom additional code> '</Custom additional code>
Function plus(A As Integer, B As Integer) As Integer Return A + B End Function
private void RunScript(object x, object y, ref object A) { } // <Custom additional code> // </Custom additional code>
public int plus(int a, int b) { return a + b; }
def plus(a,b): return a+b
Nothing visible.
49 Challenge the future
Basics of Scripting in Rhino + GH
•Rhinocommon •Rhinoscript •Grasshopper Kernel
Rhinocommon
Library of
Geometry
Operations VB.NET. C#.NET &
Python
Grasshopper Kernel
Library of Some
Special Geometry
Operations VB.NET. C#.NET &
Python
Rhionscript (with [Iron]Python)
Operations as in Rhino Command Line
Python
http://4.rhino3d.com/5/rhinocommon/ Rhinocommon SDK:
Grasshopper SDK: [Rhino]Command: GrasshopperGetSDKDocumentation
Rhinoscript Syntax SDK: [GH Python]Help: Rhinoscript syntax help
50 Challenge the future
Questions? [email protected]
• I will give you the tools we have developed so far as open source;
• You will make them better using the following libraries; • Using either Math.NET or MetaNumerics is a must; • Using Accord.NET is a big plus! • If you are using Python, use similar libraries such as SciPy • If you want to do something else let us discuss it now!