Department of Science and Technology Institutionen för teknik och naturvetenskap Linköping University Linköpings Universitet SE-601 74 Norrköping, Sweden 601 74 Norrköping

LiU-ITN-TEK-A--09/027--SE

Polygonal models from rangescanned trees

Li Qiu

2009-04-24

LiU-ITN-TEK-A--09/027--SE

Polygonal models from rangescanned trees

Examensarbete utfört i medieteknikvid Tekniska Högskolan vid

Linköpings universitet

Li Qiu

Handledare Mark DieckmannExaminator Mark Eric Dieckmann

Norrköping 2009-04-24

Upphovsrätt

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –under en längre tid från publiceringsdatum under förutsättning att inga extra-ordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat förickekommersiell forskning och för undervisning. Överföring av upphovsrättenvid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning avdokumentet kräver upphovsmannens medgivande. För att garantera äktheten,säkerheten och tillgängligheten finns det lösningar av teknisk och administrativart.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman iden omfattning som god sed kräver vid användning av dokumentet på ovanbeskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådanform eller i sådant sammanhang som är kränkande för upphovsmannens litteräraeller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press seförlagets hemsida http://www.ep.liu.se/

Copyright

The publishers will keep this document online on the Internet - or its possiblereplacement - for a considerable time from the date of publication barringexceptional circumstances.

The online availability of the document implies a permanent permission foranyone to read, to download, to print out single copies for your own use and touse it unchanged for any non-commercial research and educational purpose.Subsequent transfers of copyright cannot revoke this permission. All other usesof the document are conditional on the consent of the copyright owner. Thepublisher has taken technical and administrative measures to assure authenticity,security and accessibility.

According to intellectual property law the author has the right to bementioned when his/her work is accessed as described above and to be protectedagainst infringement.

For additional information about the Linköping University Electronic Pressand its procedures for publication and for assurance of document integrity,please refer to its WWW home page: http://www.ep.liu.se/

© Li Qiu

Abstract

3D Models of botanical trees are very important in video games, simulation, vir-tual reality, digital city modeling and other fields of Computer Graphics. However,since the early days of computer graphics, the modeling of trees has been challeng-ing, because of the huge dynamical range between its smallest and largest struc-tures and their geometrical complexity. Trees are also ubiquitous which makesit even hard to model them in a realistic way. Current techniques are limitedin that they model a tree either in a rule-based way or in an approximated way.These methods emphasize appearance while sacrificing its real structure. Recentdevelopment in range scanners are making 3D acquisition feasible for large andcomplex objects.

This report presents the semi-automatic technique developed for modelinglaser-scanned trees. First, the user draws a few strokes on the depth image planegenerated from the dataset. Branches are then extracted through the 2D Curvedetection algorithm originally developed. Afterwards, those short branches areconnected together to generate the skeleton of the tree by forming a MinimumSpanning Tree(MST). Finally, the geometry of the tree skeleton is produced usingallometric rules for branch thickness and branching angles

v

Acknowledgments

I wish to express sincere appreciation to Prof. Baoquan Chen, from ShenzhenInstitute of Advanced Technology(SIAT), Chinese Academy of Sciences(CAS), forproviding me with this thesis topic and his guidance throughout the project. Iwould also like to acknowledge Doctor Zhanglin Cheng for giving me the back-ground information on Branches Reconstruction from Range Image and supplyingme dataset. Furthermore, I would like to thank all the staff in Visual Comput-ing Center(VCC) of SIAT for many exciting discussions on Point-Based Graphics.In particular, I want to thank Doctor Jun Luo for the discussions on Computa-tional Geometry and Yotam Livny, from Ben-Gurion University of Israel, for thediscussions on 2D curve detection.

I would like to thank all those who gave me advice, supported through thestudy of the programme of Advanced Computer Graphics. They are(in no partic-ular order): Mark E Dieckmann, Aida Vitoria, Tomas Otby, Anders Ynnerman,Haraldur Haralds, Richard Khoury, Willem Frishert, Chiyu Wang.

vii

Contents

1 Introduction 1

1.1 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Background 3

2.1 Rule-based Plant Modeling . . . . . . . . . . . . . . . . . . . . . . 32.1.1 L-system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.2 Tree modeling using L-system . . . . . . . . . . . . . . . . . 4

2.2 Sketch-based Tree Modeling . . . . . . . . . . . . . . . . . . . . . . 52.3 Image-based Plant Modeling . . . . . . . . . . . . . . . . . . . . . . 62.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 RANSAC 9

3.1 RANSAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Efficient RANSAC . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.3 2D Curve Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3.1 Lagrange Interpolating Curve . . . . . . . . . . . . . . . . . 113.3.2 Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Modeling of Laser-Scanned Trees 13

4.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 Sketching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.2.1 Rendering Issues . . . . . . . . . . . . . . . . . . . . . . . . 144.2.2 Sketching . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.3 Connecting Branches . . . . . . . . . . . . . . . . . . . . . . . . . . 164.3.1 Minimum Spanning Tree(MST) . . . . . . . . . . . . . . . . 164.3.2 Connecting Branches using MST . . . . . . . . . . . . . . . 17

4.4 Mesh Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.4.1 Tree Allometry . . . . . . . . . . . . . . . . . . . . . . . . . 174.4.2 Skeleton Mesh . . . . . . . . . . . . . . . . . . . . . . . . . 18

5 Results 19

ix

x Contents

6 Conclusions and Future Work 23

6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6.2.1 Image Segmentation Techniques . . . . . . . . . . . . . . . 236.2.2 3D Ransac Branch Detection . . . . . . . . . . . . . . . . . 24

Bibliography 25

Chapter 1

Introduction

This chapter gives an overview of the subject and the objective to achieve for thisdiploma work. It also outlines the layout of this thesis.

1.1 Problem Description

3D Models of botanical trees are very important in video games, simulation, virtualreality, digital city modeling and other fields of Computer Graphics. However,since the early days of computer graphics, modeling trees is challenging because ofits large scale of structure and geometrical complexity. Trees are also ubiquitouswhich makes it even hard to model in a realistic way. Current techniques limitthe modeling of tree either in a rule-based way or in an approximated way. Thesemethods emphasize trees’ appearance while sacrificing its real structure. Recentdevelopments in range scanners are making 3D acquisition feasible for large andcomplex objects like trees. This report presents the semi-automatic technique ofmodeling laser-scanned trees.

1.2 Objective

The objective of the project presented in this thesis is to produce polygonal modelswith fine details from laser-scanned trees. The tree data acquired by 3D scanning isusually in the form of point cloud, which is a set of vertices in a three-dimensionalcoordinate system, measured from a large number of points on the surface of ob-jects. Due to the lack of topology information, point clouds are generally notdirectly rendered in most 3D applications before converted to polygon models.Traditional techniques for converting a point cloud to a polygon mesh include De-launay triangulation, Marching triangles [1] and the Ball-Pivoting algorithm [2].Unlike buildings and manufactured part which have relatively simple geometry,trees are more complex and micro-structured. Moreover, the small branches atfoliage region are often missing because of limited scanning resolution and occlu-sion from leaves and other branches. The challenge is to reproduce the models as

1

2 Introduction

accurate as possible as real trees.

1.3 Structure of the Thesis

The report is organized as follows: Chapter 2 provides more details on the back-ground and describes most recent developments of tree modeling in the field ofcomputer graphics. Chapter 3 gives a brief introduction of RANSAC and our ex-tension with it. Chapter 4 presents our approach of building polygon model fromlaser-scanned trees. The results are shown and discussed in chapter 5, followed bythe conclusions and future work presented in chapter 6.

Chapter 2

Background

This chapter gives a brief discussion of previous work and the recent advancesin tree modeling. The techniques for modeling bushes and trees can be roughlyclassified into rule-based, sketch-based and image-based plant modeling. Rule-based system defines a set of rules and symbols to iteratively generate a stringwhich will be interpreted as self-similar fractals at the final stage. Sketch-basedtree modeling usually requires user input e.g. a few strokes to reconstruct the 3Dbranches. Sketch-based systems provide a more intuitive way of generating plantmodels. Image-based tree modeling uses images to help generate 3D models.

2.1 Rule-based Plant Modeling

2.1.1 L-system

Astrid Lindenmeyer [3] first introduced L-system as a formalism for simulating thedevelopment of multicellular organs in terms of division, growth, and individualcells. An L-system or Lindenmayer system is a formal grammar(a set of rulesand symbols). L-systems can also be used to generate self-similar fractals such asiterated function systems [4]. L-system is competent for describing self-similarityand fractal-like forms due to its recursive nature [4]. By increasing the recursionlevel, the form slowly grows and becomes more complex. L-system is also popularin generating artificial life.

L-system can be defined as a tuple

G = {V, S, ω, P} (2.1)

where:

• V(the alphabet): is a set of symbols containing elements that can be replaced(variables).

• S: is a set of symbols containing elements that remain fixed (constants).

3

4 Background

Figure 2.1. ’Weeds’, generated using an L-system in 3D

• ω: (start, axiom or initiator): is a string of symbols from V defining theinitial state of the system.

• P: is a set of production rules or productions defining the way variables can bereplaced with combinations of constants and other variables. A productionconsists of two strings - the predecessor and the successor.

There are several extensions to the basic L-systems. Parametric L-systemallows the passing of arbitrary numerical information from parents to children. Ifa rule depends on its neighbors, it is called context-sensitive L-system; otherwise itis called context-free L-system. If there is exactly one production for each symbol,it is termed a deterministic L-system. If there are several, and each is chosen witha certain probability, then it is said to be a stochastic L-system.

L-system does not actually produce any geometry but a string contains sym-bols. Visual output comes from the mapping of the string into visual domain. Onesuch mapping is to interpret the strings as instructions to a turtle, for example,the symbol F moves the turtle forward and + rotates the turtle in certain degrees.With this mapping, the L-system can produce a large variety of squiggly lines.Since we want to use the L-system to model plants, we may map those symbolsinto cylinders and polygons to represent branch and leaves respectively. Figure 2.1shows a few tree models generated by L-system with different levels of recursion.

2.1.2 Tree modeling using L-system

Prusinkiewiczy et al. [5] gave a systematic description for using L-system to modelplants. The paper revisited basic mechanisms that control plant development: cel-lular descent, endogenous interaction represented by context-sensitive L-systems.Several most recent models developed at that time were also surveyed. Parame-terized L-systems were proposed in [6] [7] [8].

Xfog [9] is a combination of a rule-based approach and the graph-based objectinstancing paradigm. There are generally three categories of components which

2.2 Sketch-based Tree Modeling 5

Figure 2.2. Computing the depth information for branches with the 2D distancefield [10]

encapsulate data and algorithm for generating plant elements. One group of com-ponents creates graphical primitives like stems, twigs and leaves. The multiplyingcomponent connects other components by linking the component prototypes ina directed graph called the prototype graph, or p-graph. And the third appliesparameters to the global modeling. A component instance called i-tree is builtwhen the system traverses the p-graph.

2.2 Sketch-based Tree Modeling

Sketch-based systems provide a intuitive way of developing plant models. Okabeet al.’s system [10] reconstruct tree branches from 2D drawn sketches in differentviews and using the assumption that trees spread their branches so that the dis-tances between branches are as large as possible. The user begins to model a 3Dtree by sketching simple 2D branches using a mouse or pen tablet. The systemthen generates 3D tree geometry after the user clicks “3D” button. The user canadd or remove branches manually by using the editing interface. The user canalso add more child branches to a parent branch using existing branches as exam-ple. Leaves are arranged around branches by following typical leaf arrangementpatterns. Finally, branches are propagated. The task to create a 3D tree from2D sketch is fulfilled by projecting all branches to the ground and constructing a2D distance field. Distances are then computed from the projected branches toeach pixel(see Figure 2.2). However, the projection of 3D branches onto the screenshould fit the 2D sketch. This is done by restricting the search within a 3D hull,which is constructed by simply sweeping a circle along the 2D convex hull andenlarging it by a constant value.

Ijiri et al.’s system [11] is an L-system-based sketch system. The system con-sists of two panels: a control pane and a creation pane. In the control pane, theuser manipulates the control points to specify the rules and parameters. Prede-fined generating rules are also available in the system. After generating the rules

6 Background

Figure 2.3. Sketch-Based Tree Modeling Using Markov Random Field [12]

and parameters, the user draw another stroke representing the central axis in thecreation pane. The system then interprets the drawn stroke and applies the spec-ified rules at the same time the recursion process occur until it reaches the depthof recursion specified by the user.

Chen et al. [12] proposed a method to formulate the tree reconstruction prob-lem as Markov random field. In their system, the user begin to model the treewith a few strokes of branches. The closest matched specimen is then selectedfrom a predefined database which contains typical tree exemplars and their asso-ciated global parameters. Assuming that the sketch is drawn under orthographicprojection, the problem of reconstructing the tree model is reduced as evaluatingthe depth of each drawn branch. Considering each branch segment as a node andits depth as a variable, it becomes dealing with Markov tree. Additionally, explic-itly introducing additional nodes representing the relationships between neighbornodes producing a factor graph. Solving the factor graph reconstructs the 3Dshape of the drawn branches. Figure 2.3 shows the result of their system.

2.3 Image-based Plant Modeling

Rather than requiring the user to specify the model manually, Image-based ap-proaches use a few images as input to generate 3D models. A purely image-basedmodeling approach is described by Reche et al. [13]. Their system adopt a directvolumetric rendering model based on the observation that at a certain distance,each pixel in a photograph of a tree is blended projection of leaves, branches andbackground at that direction. First, opacity values of the tree volume are esti-mated from calibrated digital photographs by using alpha-mattes extracted fromthe photographs. Then, a billboard with view-dependent texture is assigned toeach grid cell. Finally, volumetric rendering approach is deployed to represent andrender the tree.

Tan et al. [14] proposed another method to reconstruct the 3D model from

2.3 Image-based Plant Modeling 7

Figure 2.4. Tan [14]’s image based tree modeling system

multiple images which captured by a hand-held camera. The system extract pointcorrespondences and ran structure from motion on them to recover camera pa-rameters and 3D point cloud by using standard computer vision techniques. Oncethe camera poses and 3D point cloud have been obtained, the system partitionthe points into branches and leaves according to their color and position. The re-construction of the visible branches starts from constructing a graph to each pointcluster. The user initiate a 3D point of the primary branch and add another branchcluster to it iteratively. After recovering the visible branches, the system recoverthe occluded branches by replicating the reconstructed parts. Finally, leaves areadded and populated. Figure 2.4 shows an overview of the system.

Neubert el al. [14] presented a new image-based modeling method for 3D treegeometry from several photographs with limited user intervention. For each giveninput image, their system first separates the tree from background and then 2D at-tractor graph is computed either automatically by using Livewire approach [15] orfrom user drawn strokes. Secondly, back-projection is applied to the input imagesto compute the density volume(see Figure 2.5), a voxel-model of the tree volumewith each voxel containing a density estimate of the tree’s biomass. Particles arethen positioned in proportion to the density values. 2D attractor graphs are usedto create direction fields which provide direction vectors for each image plane.Combining these vectors for all image planes yields vectors which direct the flowof particles. Tracing the flow of the particles produces the main skeleton of thetree. Branch thickness and branching angles are estimated using allometric rules.

Multiple images are used to generate the tree models in methods mentionedabove while single image is used in Tan. et al. [16]’s system. Given a single imageof a tree, the user draws strokes on the image to create a tree model. At least twostrokes are needed to complete the tree model: the first crown stroke around thetree crown to mark up the leaf region, the second starting at the root to mark upthe main trunk. Additional strokes are optional to refine the tree model. Afteruser draws strokes, foliage is extracted from the closed region by the first crownstroke. A Gaussian mixture model(GMM) [17] is computed for the pixel RGBvalues and graph-cut based extraction [18] is deployed to detect leaf pixels. Totrace the visible branches, branch appearance model and non-branch appearancemodel are computed. With these two models, a circle along the branch stroke

8 Background

(a) (b)

Figure 2.5. Estimating the tree density: (a)voxel grid; (b)density values for one imageplane [14]

is moved from bottom to up to recover the visible branches. Finally, the systemconverts the 2D branches and grows the tree in a similar way as that in [11].

2.4 Conclusion

Parametric L-system are able to provide the user with a few parameters. However,they are usually restricted to produce a limited number of forms. To achieve aspecific species or even a particular tree, many parameters have to be adjustedand kept which is impractical to the users. Image-based methods have the bestpotential for producing realistic-looking plants [12]. While they emphasis on theappearance of the tree, their approach does not recover explicit geometries ofthe branches and leaves. Exceptions are approaches that apply Structure FromMotion(SFM), a standard method finding the three-dimensional structure by an-alyzing the motion of an object over time, to recover 3D point cloud from imagesequences.

Chapter 3

RANSAC

This chapter introduces RANSAC, a shape detection method that extracts shapesby randomly drawing minimal sets from the data with noise and outlier, and itsvariant, Efficient RANSAC which is more useful in an interactive setting. Weextended Efficient RANSAC to support 2D curve detection for our application.

3.1 RANSAC

RANSAC, short for RANdom SAmple Consensus is an iterative method to es-timate parameters of a mathematical model from observed data which containsnoise and outliers. In this procedure, a basic assumption is that the data con-sists of inliers, i.e., data whose distribution fits some set of model parameters, andoutliers, data that can not be explained by the model(see Figure 3.1):

Given a set of observed data values, a parametrized model which can explainthe observations and some confidence parameters, RANSAC finds the optimalmodel(s) by iteratively selecting a random subset of the original data. These dataare hypothetical inliers and this hypothesis is tested as follows:

1. A model is fitted to the hypothetical inliers, i.e. all free parameters of themodel are reconstructed from the data set.

2. All other data are then tested against the fitted model and, if a point fitswell to the estimated model, also considered as a hypothetical inlier.

3. The estimated model is reasonably good if sufficiently many points have beenclassified as hypothetical inliers.

4. The model is reestimated from all hypothetical inliers, because it has onlybeen estimated from the initial set of hypothetical inliers.

5. Finally, the model is evaluated by estimating the error of the inliers relativeto the model.

This procedure is repeated a fixed number of times T until we find the themodel with lowest error.

9

10 RANSAC

(a) (b)

Figure 3.1. (a):A data set with many outliers for which a line has to be fitted; (b):Fitted line with RANSAC, outliers have no influence on the result

3.2 Efficient RANSAC

Ransac can estimate the parameters with a high degree of accuracy even whensignificant amount of outliers appears in the data set. However, there is no upperbound number of iterations it takes to compute these parameters. Moreover,the datasets are often huge, which implies that the efficiency of the algorithm is ofutmost importance to enable interactivity. In our implementation, we extended R.Schnabel et. al [19]’s Efficient RANSAC algorithm to support 2D Curve detection.Algorithm 1 shows an overview of Efficient RANSAC algorithm.

The complexity of RANSAC is dominated by two major factors: The numberof minimal sets that are drawn and the cost of evaluating the score for everycandidate shape. The number of iterations can be determined by the probabilitythat the best possible shape is indeed detected. Consider a point cloud P of size Nand a shape Ψ therein consisting of n points, k the size of a minimal set required todefine a shape candidate, the probability of any k points leading to an appropriate

3.3 2D Curve Detection 11

candidate is:

P (n) =(n

k)

/

(

N

k

)

≈( n

N

)k

(3.1)

The probability of detecting an appropriate shape after s candidates have beendrawn equals the complementary of s consecutive failures:

P (n, s) = 1− (1− P (n))s (3.2)

Thus, the number of candidates T required to detect shape Ψ with a probabilityP (n, T ) ≥ Pt:

T ≥ln (1− pt)

ln (1− P (n))(3.3)

From formula 3.3, we can see that the runtime complexity is directly relatedto the hit rate of finding good sample sets.

Shapes are usually local phenomena, which means that the probability of twopoints close to each other belong to the same shape is higher. Exploiting this factincreases the probability of drawing minimal sets belonging to the same shape.The Efficient RANSAC algorithm uses an octree to establish spatial proximitybetween samples(see R. Schnabel et. al [19]). Initially, choose points for the firstsample p1 among all points. Then, a cell C is randomly chosen from any level ofthe octree such that p1 ∈ C, and the other k− 1 samples are only chosen from C.The procedure can be expressed by a new probability Plocal (n):

Plocal (n) = P (p1 ∈ Ψ)P (p2 · · · pk ∈ Ψ|p2 · · · pk ∈ C) (3.4)

Assuming that there exists a C which contains half of its points supportingshape pi, the probability of finding a good C in depth d is:

Plocal(n) =n

Nd2k−1(3.5)

3.3 2D Curve Detection

Our application of RANSAC to tree modeling is to automatically extract a curve(branch)from user-sketched data.

3.3.1 Lagrange Interpolating Curve

Since our problem is to find the curve passes through most of the discrete pointsin 2D space, we choose to use Lagrange interpolating curve [20] justified by severalreasons. First, Lagrange curve only requires a few points as input, which can beobtained by simply sampling from given data source. Secondly, the degree of thepolynomial is usually low.

Given a set of k + 1 data points

12 RANSAC

(x0, y0) , · · · , (xk, yk) (3.6)

where none of them are identical, the interpolation polynomial in the Lagrangeform is a linear combination

L(x) :=k∑

j=0

yj lj(x) (3.7)

of Lagrange basis polynomials

lj(x) :=k∏

i=0,i 6=j

x− xixj − xi

=(x− x0)

(xj − x0)· · ·

(x− xj−1)

(xj − xj−1)

(x− xj+1)

(xj − xj+1)· · ·

(x− xk)

(xj − xk).

(3.8)

3.3.2 Scoring

The scoring function σp measures the quality of a given curve. In our case, thepoints on a curve may not be successive while disconnected with several partsbecause a branch may be occluded by other branch or leaves(see Figure 4.3). Tomeasure the score of a candidate curve, we use only the number of points that fallwithin an ǫ-band around the curve, that is, ǫ close to the curve.

More formally, given a curve Ψ whose fidelity is to be evaluated, σp is definedas follows:

σp(Ψ) = |PΨ| (3.9)

The number of points in PΨ is defined in the following step:

PΨ = {p|p ∈ P ∧ d(Ψ, p) < ǫ} (3.10)

where d(ψ, p) is the euclidean distance between the point p and the curve Ψ.

Chapter 4

Modeling of Laser-ScannedTrees

This chapter first gives an overview of our approach of modeling tree from pointclouds. Each step is then detailed in the following subsections. Our approach firstrequire the user to draw a few strokes on the depth image plane generated from thedataset. Branches are then extracted through the 2D Curve detection algorithm.Afterwards, those short branches are connected together to generate the skeletonof the tree by forming a Minimum Spanning Tree(MST). The geometry of the treeskeleton is produced using allometric rules for branch thickness and branchingangles

4.1 System Overview

Given the point cloud of a laser-scanned tree, we start the polygon modelingprocess by first constructing the skeleton, which is a good representation of treestructure. The geometry of the skeleton is produced by using allometric theory.Finally, leaves are added to nearby branches. The outline below also reflects thefurther structure of this chapter:

1. Sketching: The user draws a few strokes to represent the branches.

2. Skeleton Generation: Based on the strokes drawn by user, the system au-tomatically search the branch from the 3D point cloud. Those branchesare resampled and connected together by forming a Minimum SpanningTree(MST).

3. Mesh Generation: After the full skeleton is generated, the thickness of eachbranch is determined by using botanical rules. Once the radius has beenestimated for each node, mesh is generated by connecting rings paced ateach node.

13

14 Modeling of Laser-Scanned Trees

Figure 4.1. Points scanned from a oblique branch

4.2 Sketching

To reconstructed the skeleton of the tree, a variety of methods are surveyed. Curveextraction [21] [22] [23] [24] and curve tracing [25] [26] both involves classifyingpixels as being curve points or not depending whether they lie on some curve inthe image and linking curve points to form curves. The difference between curveextraction and curve tracing is that curve extraction emphasizes on the extractionwhile curve tracing on the identification or labeling a curve. Pattern recogni-tion techniques such as matching-filters [27], ridge-based approaches [27], regiongrowing [27], multi-scale approaches [27], skeleton-based approaches [27], level setmethods [27], tracking-based approaches [27] et al., are popular approaches inmedical segmentation.

Due to limited scanning resolution and occlusion from leaves and other branches,improper classifying and linking of curve(branch) are prone to happen. Moreover,small branches and leaves in the crown are often irregular and chaotic, and whenthe branch is scanned from a glancing angle, very sparse points are sampled fromit(see Figure 4.1). To alleviate the influence of these problems and produce a goodmain skeleton, the system requires user to extract the skeleton.

4.2.1 Rendering Issues

We prefer rendering the point cloud with scanning image plane rather than doingit with standard 3D rendering pipeline since there are big holes as zooming in tolook into the details of the structure of the tree for user interaction. We also colorthe points with its depth such that it is easy to see the occlusion of branches fromeach other(see Figure 4.2 and Figure 4.3).

4.2 Sketching 15

Figure 4.2. Rendering point cloud for user interaction. Top right: render the points aspoint sprites. Bottom right: render the points in scanning image plane.

Figure 4.3. Depth view of the tree

16 Modeling of Laser-Scanned Trees

Figure 4.4. Curve detection using RANSAC

4.2.2 Sketching

The user draws a few strokes on the image by moving the mouse cursor and holdinga button to mark out branches and those occluded by other branches that are infront of them. During this process, a band of pixels are selected around the currentstroke and projected to the curve drawn. Next, the 2D curve detector will findout the best curve representing the branch(see Figure 4.4).

4.3 Connecting Branches

4.3.1 Minimum Spanning Tree(MST)

In the mathematical field of graph theory, given a connected, undirected graph G,a spanning tree T is a tree composed of all the vertices and some of the edges ofG [28]. A single graph can have many different spanning trees. If the graph G is

4.4 Mesh Production 17

Figure 4.5. The minimum spanning tree of a planar graph. Each edge is labeled withits weight, which here is roughly proportional to its length.

weighted, the weight of the spanning tree is the sum of the weights of the edges inthe spanning tree. A minimum spanning tree(MST) or minimum weight spanningtree is then a spanning tree with weight less than or equal to the weight of everyother spanning tree(see Figure 4.5).

4.3.2 Connecting Branches using MST

The weighted complete graph G = {V,E} is built by taking each branch as a nodeand connecting it to the other branches. The weight for each edge is computedusing distance of two branches. The distance of two branches is defined as the min-imum 3D Euclidean distance of the supported nodes on those branches penalizedby the deviation of the ends:

d(p, q) =

d3D(p, q), if p ∈ B1 and q ∈ B2 and (p 6= E1 or q 6= E2)

d3D(p, q)/sin(α ∗ 0.5), otherwise.(4.1)

where B1, B2 are supported node sets of two branches, E1, E2 are two ends.The supported node is defined as the node on the curve that can be connected tothe other. The weight on each edge reflects the likelihood that the two branchesbeing connected in the real tree structure.

4.4 Mesh Production

4.4.1 Tree Allometry

There are many models formalizing the size and structure of trees. In 15th century,Leonardo da Vinci stated that “all the branches of a tree at every stage of its height

18 Modeling of Laser-Scanned Trees

when put together are equal in thickness to the trunk” [29]. Pipe Model [30] [31]assumes that trees are composed of bundles of unit pipes and each unit pipesupplies nutrients to leaves. Thus, the sum of the cross-sectional areas of all childbranches is equal to that of the parent branch. However, this model doesn’t obeymechanics. In Murray Model, the sum of the cross sectional area gets greater ateach split:

r2.49p = r2.49

c1 + r2.49c2 (4.2)

where rp is the radius of the parent branch, and rc1 and rc2 the radii of twochildren if a binary split. The angle between the directions of parent and eachchild can be described by following equation:

cosθ =r4p + r4

c − (r3p − r

3c )

4/3

2r2pr

2c

(4.3)

In our implementation, we use Murray’s law for branch splitting, and a sim-plified WBE model for non-splitting nodes. The same technique was used by Xuet al. [32]

4.4.2 Skeleton Mesh

Once the radius has been estimated for each node in the skeleton, a ring is placedaround each node perpendicular to its direction. At rendering time, a cyclic barkare applied to the trunk and branches. According to the density defined for differ-ent leaf location, we assign a quad with different number of leaves to them. Quadsize is proportionally adjusted based on the number of leaves.

Chapter 5

Results

This chapter shows the results of the modeling pipeline. The process was im-plemented on a PC with Intel Core 2 2.33GHz and 2.0G of main memory. Theconstructed tree models are rendered using DirectX 9.0 on an nVidia GeForce8800GT graphics card with 512MB video memory. The point clouds are scannedby Riegl Inc’s LMS-Z360I 3D imaging sensor. Textured are synthesized from pho-tographs of the scanned trees.

Figure 5.1 and Figure 5.2 demonstrates the most difficult case of a tree modelconstructed from a single point cloud (114,997 points). Even though the datafrom only one scan is very incomplete, our system is still capable of generatingpolygonal model that plausibly recreates the original tree. It only takes a fewminutes to generate the tree model. Rendering of the models can be done in realtime to provide timely feedback to the user.

19

20 Results

Figure 5.1. Mesh model rendered of the scanned tree

21

Figure 5.2. Rendered mesh model from another view

Chapter 6

Conclusions and FutureWork

This report has presented a method of producing polygonal models from laser-scanned trees. The whole process only takes a few minutes to create a plausibletree directly from the point cloud. User can also modify the rendered result in realtime. This chapter discusses limitation of our method and mentions future work.

6.1 Conclusion

This report has presented a semi-automatic method of producing polygonal modelsfrom laser-scanned trees based on user sketching. A 2D curve detection approachwas proposed based on Efficient Ransac algorithm. Allometric theory was used toestimated the radii of tree skeleton. Our model approximates real tree structureas accurate as possible.

6.2 Future Work

There are several strategies which can be combined or used separately to makethe system more automatic and efficient. Image segmentation techniques andcurve parsing techniques can be used to extract curvilinear structures or region ofinterest from the generated depth image. Ransac can also be used to detect thebranch in 3D space directly.

6.2.1 Image Segmentation Techniques

Curve extraction [21] [22] [23] [24] and curve tracing [25] [26] both involves clas-sifying pixels as being curve points or not depending whether they lie on somecurve in the image and linking cure points to form curves. Pattern recognitiontechniques such as matching-filters, ridge-based approaches, region growing, multi-scale approaches, skeleton-based approaches, level set methods, tracking-based

23

24 Conclusions and Future Work

Figure 6.1. Result from Tu [33]’s image parsing algorithm

approaches, Graph-cut approaches et al, are popular approaches in image segmen-tation. Tu [33] introduced an algorithm for parsing images into middle level visionrepresentations - regions, curves, and curve groups(see Figure 6.1). Adaptions ofsuch techniques are useful for extracting tree structure from single view.

6.2.2 3D Ransac Branch Detection

To make the system more automatic, detecting branches in 3D space is necessary.However, from our experiments, this process remains the difficulty for the followingreasons. First, real tree branches are usually chaotic and irregular, which makesthem impractical to fit curves. Secondly, defining single scoring function that worksfor all the branches is non-trivial, especially for those scanned from a glancing angleand those occluded by others.

Bibliography

[1] A. Hilton, A. J. Stoddart, J. Illingworth, and T. Windeatt. Marching tri-angles: range image fusion for complex object modelling. Image Processing,1996. Proceedings., International Conference on, 1:381–384 vol.2, 1996.

[2] F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, and G. Taubin. Theball-pivoting algorithm for surface reconstruction. IEEE Transactions on Vi-sualization and Computer Graphics, 5(4):349–359, 1999.

[3] Aristid Lindenmayer. Mathematical models for cellular interactions in devel-opment ii. simple and branching filaments with two-sided inputs. Journal ofTheoretical Biology, 18(3):300–315, March 1968.

[4] David S. Ebert, Kenton F. Musgrave, Darwyn Peachey, Ken Perlin, andSteven Worley. Texturing & Modeling: A Procedural Approach, Third Edi-tion (The Morgan Kaufmann Series in Computer Graphics). Morgan Kauf-mann, December 2002.

[5] Przemyslaw Prusinkiewicz, Mark Hammel, Jim Hanan, and Radom R Mech.L-system: from the theory to visual models of plants. Machine Graphics andVision, pages 365–392, 1996.

[6] Jason Weber and Joseph Penn. Creation and rendering of realistic trees. InSIGGRAPH ’95: Proceedings of the 22nd annual conference on Computergraphics and interactive techniques, pages 119–128, New York, NY, USA,1995. ACM.

[7] Phillippe de Reffye, Claude Edelin, Jean Françon, Marc Jaeger, and ClaudePuech. Plant models faithful to botanical structure and developmentr. InSIGGRAPH ’88: Proceedings of the 15th annual conference on Computergraphics and interactive techniques, pages 151–158, New York, NY, USA,1988. ACM.

[8] Peter E. Oppenheimer. Real time design and animation of fractal plants andtrees. In SIGGRAPH ’86: Proceedings of the 13th annual conference onComputer graphics and interactive techniques, pages 55–64, New York, NY,USA, 1986. ACM.

25

26 Bibliography

[9] Bernd Lintermann and Oliver Deussen. Interactive modeling of plants. IEEEComput. Graph. Appl., 19(1):56–65, 1999.

[10] Makoto Okabe, Shigeru Owada, and Takeo Igarashi. Interactive design ofbotanical trees using freehand sketches and example-based editing. ComputerGraphics Forum, 24(3):487–496, 2005.

[11] Takashi Ijiri, Shigeru Owada, and Takeo Igarashi. The sketch l-system: Globalcontrol of tree modeling using free-form strokes. Smart Graphics, pages 138–146, 2006.

[12] Xuejin Chen, Boris Neubert, Ying-Qing Xu, Oliver Deussen, and Sing BingKang. Sketch-based tree modeling using markov random field. ACM Trans.Graph., 27(5):109, 2008.

[13] Alex Reche Martinez, Ignacio Martín, and George Drettakis. Volumetricreconstruction and interactive rendering of trees from photographs. ACMTrans. Graph., 23(3):720–727, 2004.

[14] Boris Neubert, Thomas Franken, and Oliver Deussen. Approximate image-based tree-modeling using particle flows. page 88, New York, NY, USA, 2007.ACM.

[15] Artur Chodorowski, Morgan Langille Ulf Mattsson, and Ghassan Hamarneh.Color lesion boundary detection using live wire. SPIE Medical Imaging,5747:1589–1596, 2005.

[16] Ping Tan, Tian Fang, Jianxiong Xiao, Peng Zhao, and Long Quan. Singleimage tree modeling. In SIGGRAPH Asia ’08: ACM SIGGRAPH Asia 2008papers, pages 1–7, New York, NY, USA, 2008. ACM.

[17] H. Permuter and J. Francos. Gaussian mixture models of texture and colourfor image database retrieval. In in Proc . ICASSP, pages 25–88, 2003.

[18] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut": inter-active foreground extraction using iterated graph cuts. ACM Trans. Graph.,23(3):309–314, August 2004.

[19] Ruwen Schnabel, Roland Wahl, and Reinhard Klein. Efficient ransac forpoint-cloud shape detection. Comput. Graph. Forum, 26(2):214–226, 2007.

[20] Richard L. Burden and J. Douglas Faires. Numerical analysis: 4th ed. PWSPublishing Co., Boston, MA, USA, 1989.

[21] J Canny. A computational approach to edge detection. IEEE Trans. PatternAnal. Mach. Intell., 8(6):679–698, 1986.

[22] C. Heipke, C. Steger, and R. Multhammer. A hierarchical approach to auto-matic road extraction from aerial imagery. In in Integrating PhotogrammetricTechniques with Scene Analysis and Machine Vision II, pages 222–231, 1995.

Bibliography 27

[23] A. Baumgartner, C. Steger, C. Wiedemann, H. Mayer, W. Eckstein, andH. Ebner. Update of roads in gis from aerial imagery: Verification and multi-resolution extraction. International Archives of Photogrammetry and RemoteSensing, 1996.

[24] David Donoho. Applications of beamlets to detection and extraction of lines,curves and objects in very noisy images. In Nonlinear Signal and ImageProcessing (NSIP), 2001.

[25] Carsten Steger. An unbiased detector of curvilinear structures. TechnicalReport FGBV–96–03, Forschungsgruppe Bildverstehen (FG BV), InformatikIX, Technische Universität München, July 1996.

[26] Thomas W.Parks Karthik Raghupathy. Improved curve tracing in images.IEEE International Conference on Signal Processing, 2004.

[27] Cemil Kirbas and Francis Quek. A review of vessel extraction techniques andalgorithms. ACM Comput. Surv., 36(2):81–121, June 2004.

[28] Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and CliffordStein. Introduction to Algorithms, Second Edition. The MIT Press, Septem-ber 2001.

[29] Leonardo da Vinci. The Notebooks of Leonardo da Vinci (Oxford World’sClassics). Oxford University Press, USA, October 1998.

[30] Kichiro SHINOZAKI, Kyoji YODA, Kazuo HOZUMI, and Tatuo KIRA. Aquantitative analysis of plant form-the pipe model theory : I.basic analyses.Japanese J. of Ecology, 14(3):97–105, 19640601.

[31] K. Shinozaki, K. Yoda, K. Hozumi, and T. Kira. A quantitative analysis ofplant form – the pipe model theory ii. further evidence of the theory andits application in forest ecology. Japanese Journal of Ecology, 14(4):133–139,August 1964.

[32] Hui Xu, Nathan Gossett, and Baoquan Chen. Knowledge and heuristic-basedmodeling of laser-scanned trees. ACM Trans. Graph., 26(4):19, 2007.

[33] Zhuowen Tu and Song-Chun Zhu. Parsing images into regions, curves, andcurve groups. Int. J. Comput. Vision, 69(2):223–249, 2006.