PIECEWISE LINEAR CYLINDER MODELS FOR 3-DIMENSIONALAXON SEGMENTATION IN BRAINBOW IMAGERY

Erhan Bas, Deniz Erdogmus

Cognitive Systems Laboratory, ECE Department, Northeastern University, Boston, MA 02115, USA

ABSTRACTGeneralized cylinder shapes are ubiquitous in biological sys-tems and image processing techniques to identifies thesetubular objects in 3D from biomedical imagery in vari-ous modalities is a general problem of interest. One suchstructure that exhibits branching tubular forms are neuronalnetworks; specifically, recent developments in microscopyimaging technology allow researchers to acquire massiveamounts of 3D color images of neural structures that needto be tracked in 3D to extract structure for the purpose ofstudying function. In this paper, we propose a piecewise lin-ear generalized cylinder tracing algorithm that exploits bothedge and color information in order to automatically traceaxons of neurons in Brainbow imagery. Results indicate thatthe proposed method can successfully trace multiple axons indense neighborhoods.

Index Terms— Locally linear generalized cylinders,axon segmentation, Brainbow

1. INTRODUCTION

Generalized cylinder shaped objects are ubiquitous in 3Dbiomedical imagery; many structures exhibit tree topologieswith cylindrical branches (e.g., vascular networks, dendritictrees, bronchia). Consequently, significant research efforthas been dedicated to the segmentation of objects with suchstructures that arise in various contexts [1, 2, 3]. Shape mod-els informed by knowledge of the geometry and physicalconstraints of the biological structures are useful in improv-ing automatic segmentation results; however, accurate shapemodels require the availability of manually segmented train-ing data, which requires immense manual labor and timecommitment to acquire. Consequently, the design of segmen-tation algorithms that can both aid the process of training dataextraction and that can then utilize the new available data toits advantage by incorporating the information as shape priorswill facilitate the development of accurate automatic segmen-tation approaches in this domain.

In general, since clustering based approaches are notsuitable for the straightforward incorporation of shape pri-ors, active contour and level set techniques that are basedon the evolution of curves or surfaces that optimize somesuitable energy function that combines multiple optimalitymeasures, one of which imposes the desired shape prior in

DE (erdogmus@ece.neu.edu) and EB (bas@ece.neu.edu) acknowledgesupport from NSF grants ECCS-0929576 and ECCS-0934506. The authorswould also like to thank Bill Bosl and Dana Brooks for valuable discussionsand to Ryan Draft for providing the Brainbow data and helping with the con-tents of Section 2.

penalty form [4, 2], through scalarization (e.g., linear combi-nation) are preferred. The designed energy function is usedto highlight object boundaries [5] and these curve and surfaceevolution methods have been shown to work well in a varietyof scenarios with careful tuning. These approaches, however,require a detailed level of topological and structural under-standing of the dataset in order to define energy functions thatwill succeed. These measures are functions of image inten-sity (e.g., gradient-norm or other edginess metric) and spatialand shape penalties. In order to highlight certain curvilinearstructures, shape filters have also been employed [6, 7]. Eige-nanalysis of the Hessian matrix of the image intensity is alsoa popular approach to identify objects (e.g., vessels), whereeigenvalue ratios can be related to local curvature [8, 7];however, determining such relationships require extensiveexperience and training data.

Brainbow images, a recently developed technique for ac-quiring 3D colored confocal microscopy imagery that depictneuronal networks in the brain, present a great opportunity forneuroscientists to study brain structure and function; however,the speed of progress is hampered by the inability to automat-ically extract structural information about the network fromthese images due to the lack of robust and reliable 3D cylin-drical object segmentation techniques for color images thatcould automatically trace axons (techniques designed for MRand CT specialize on intensity and they primarily focus onslice-based segmentation [1, 2, 3, 9]). Currently, researchersusing Brainbow technology are limited to manual segmenta-tion of images and tracing of axons to identify neural net-works; this process is clearly cumbersome and prevents thestudy of large brain sections.

In this paper, we propose an algorithm for semiautomaticsegmentation of objects with generalized cylinder shapes (ax-ons) utilizing both edge and color information without mak-ing strict shape assumptions (such as curvature bounds or pri-ors). The approach involves building piecewise linear cylin-der approximations starting from a single seed point insidethe axon given by the user. Future improvements will includeautomatic seed point identification and incorporation of infor-mation obtained from semiautomatically segmented imageryas priors. The algorithm will enable neuroscientists who ac-quire Brainbow images to extract 3D network structure withminimal labor and intervention.

2. BRAINBOW IMAGES & PROBLEM STATEMENT

The brain processes and propagates information throughthousands of structural connections within neural networks.Understanding the organization of these ’circuits’ is thoughtto be necessary to understand how such networks execute andadapt complex behaviors. Investigations of networks have

been prevented, however, due to the length of 3D projectionsof single neurons (mm’s) and the density and thinness ofstructures (

previous cylinder and current sphere must be greater than 0.9:

(pi+1, ri+1) = arg minp,r

1∑i E(si; p, r)

subject to (2)

nTi (p− p̂i+1) = 0(I− ninTi )(p− p̂i+1) < riG(mi −mS(p,r); Σi + ΣS(p,r))

G(0; 2Σi)G(0; 2ΣS(p,r))> 0.9

where, si are obtained using a 21-by-21 grid in angular co-ordinates with π/10 radian resolution in θ (longitude) andπ/20 radian resolution in φ (latitude)1, E(si; p, r) is obtainedfrom the edge map E via 3D-cubic spline interpolation, andG(x,C) = (2π)−n/2 det(C)−1/2 exp(− 12x

T C−1x), withmi & Σi being the estimated mean and covariances of theGaussian color distribution model in RGB space for the vox-els in the last cylinder (denoted by index i), and mS(p,r)& ΣS(p,r) similarly being the estimated color mean andcovariances for the voxels in the current sphere S(p; r).The mean and covariance estimates are obtained usingExpectation-Maximization (EM) with a Gaussian color den-sity model. Note that if we define the inner product betweentwo Gaussian densities Gj = G(x−mj ; Σj) for j = 1, 2 as< G1, G2 >=

∫G(x−m1; Σ1)G(x−m2; Σ2)dx, then

< G1, G2 >

(< G1, G1 >< G2, G2 >)1/2=

G(m1 −m2; Σ1 + Σ2)G(0; 2Σ1)G(0; 2Σ2)

(3)

Step 2: The direction ni+1 is identified as the largesteigenvector (whose sign is selected such that it makes a pos-itive inner product with ni) of a weighted covariance matrixW of voxel coordinate vectors for each voxel vj in a cubethat contains sphere S(pi+1, ri+1) and that has edges alignedwith the xyz-axes of the image coordinate system. The weightof each voxel is obtained from the Gaussian color model iden-tified from the voxels inside the sphere using EM as describedin step (1). Specifically:

W =∑

j

G(dj ; s2I)G(cj −mSi+1 ; ΣSi+1)djdTj (4)

where dj = vj − pi+1 is the displacement of voxel j fromthe center of the current sphere Si+1 = S(pi+1, ri+1).

3.3. Propagation

Once the circular crossection of the cylinder is identified us-ing the tuning process described above, a linear extension ofthe cylinder in the direction pointed to by ni is generated us-ing increments of length equal to the edge of one cubic voxeluntil a criterion leaves the margin determined by adaptivelyset lower and upper bounds. The criterion in particular, for a

1A better sampling strategy can be achieved by using uniformly displacedgrid samples on the sphere. Approximations can be done by solving a con-straint optimization problem with equally displaced grid locations constrainton the sphere manifold.

given cylinder of length l, is given by:

Υ(l) =∑

k

E(qk;C(pi+1, ri+1, lni))

−l∑

j

E(sj ;B(pi+1, ri+1,ni)) (5)

where C(pi+1, ri+1, lni) is a cylinder with base center pi+1,radius ri+1, with length l in the direction of ni, and qk aresamples from its surface taken on a uniform grid along the an-gle of the circular base (with π/10 rad resolution) and length(at a resolution of one voxel). Similarly, B(pi+1, ri+1,ni) isthe base of this cylinder and sj are samples taken uniformlyfrom its circular boundary (at π/10 rad sample period). Thelower and upper thresholds are selected to be proportional tothe second term in the criterion above (specifically, we haveused factors of 0.98 and 1.05).

4. RESULTS

We demonstrate selected results from the proposed method ona brainbow image set composed of 31 slices (z-direction res-olution of 64 µm) with each slice being 1024 × 1024 pixels(x & y directions at 11 µm resolution). In the preprocessingstep the images are downsampled by 3 in the x & y directionsand upsampled by 2 in the z direction yielding a voxel size of33 × 33 × 32 µm3. The resampled image stack is smoothedin 3D using a bilateral filter with Gaussian kernel with diago-nal covariance (spatial scale of 5 voxels and RGB-color scaleof 0.2). Fig. 1 shows sample axial slices of the axons aftersmoothing and a coronal slice of the edge map, E is shown inFig. 2. While all axons can be identified and traced success-fully, in Fig. 4 only a couple of selected axons are displayed.In order to emphasize weak edges, we also used the square-root of the edge values to select and iterate additional fibers.Columns of Fig. 5 show two different yz-plane slices of thebrainbow image stack where first row is the filtered image,second row shows the corresponding edge map slices, and thelast row displays the selected fiber projections on the square-root of the edge map.

5. DISCUSSION AND FUTURE WORK

The automatic identification and segmentation of fiber/vessel-shaped objects in 3D imagery is a commonly encounteredproblem. Brainbow images in particular provide a novel toolfor neuroscientists to study the structure of the nervous systemin vitro via confocal microscopy. In this paper, we presented amethod for identifying axons in 3D image sets using edge andcolor information with weak shape priors incorporated in theform of a piecewise linear generalized cylinder approximationand constraints imposed during the optimization process. Theproposed approach, at this point, aims to be semiautomatic inorder to reduce the requirement of manual labor and inter-vention, yet allow the neuroscientist to have control over theidentified axon-tree solutions. This will enable the accumu-lation of more ground-truth information about axon networkgeometries in various parts of the nervous system, thus willallow us to improve the presented algorithm in the future fur-ther by incorporating prior information that can be extractedfrom such a database.

The algorithm exploits the knowledge that each branch ofan axon is a generalized cylinder and approximates this form

Fig. 4. Selected identified axons in a given 3D region (withaxis units in voxels).

Slice 1 Slice 2

a)

b)

c)

Fig. 5. a) Different Coronal (yz-plane) slices of a samplebrainbow image stack b) Corresponding edge values c) Pro-jected fibers on the image slice showing the square-root of theedge values.

using piecewise linear cylinders locally. This approach elimi-nates the complicated global optimization required for activeshapes or level sets that minimize energy functions over thewhole volume or surface. The local cylinder fitting approachresults in a relatively simpler energy minimization type prob-lem and the piecewise approximation does not lose its abil-ity to model highly complex axon trajectories. Continuity isguaranteed by imposing constraints on the search space forconsecutive cylinder pieces and the final smooth generalizedcylinder can be approximated by a smooth interpolation tech-nique such as cubic spline in 3D.

While the algorithm utilizes the edge information to fitindividual local cylinders, it employs color model continuityto prevent the local cylinders from jumping from one axonto another in the vicinity. In fitting the Gaussian color den-sity model in RGB coordinates, we used the EM algorithminstead of sample mean and sample covariance estimates forthe mean and covariance. It is possible for some voxels inthe given cylinder or sphere to have a multimodal color dis-tribution (e.g. background, target axon, and a neighbor axonwith a different color), thus sample statistics will be severelyinfluenced by these additional modes, which in turn could cre-ate problems in determining the local fiber direction using thelargest eigenvector of the weighted covariance matrix duringthe tuning step or the length of the fiber that is calculated inthe propagation step. Therefore propagation purely based onthe edge values is not preferred and color values of the voxels

are utilized.Problem formulation favors uniformly distributed strong

edges having larger edge values which lead to easier segmen-tation of fibers. Depending on the location and the color, edgevalues vary throughout the dataset; thus proposed method as-sumes local thresholding strategy to adaptively select upperand lower bounds in the propagation step. Moreover, in orderto emphasize the weak edges having smaller edge values, thesquare-root of the edge values are used. Choice of using thesquare-root of the edge values is heuristic and future work in-cludes image enhancement strategies that will enable us to getmore uniform edge value distributions throughout the dataset.

A desirable feature in semi-automatic segmentation is theability of the user to intervene with the solution in a flexibleand local manner. The proposed method gives the opportunityto the user to intervene in the initialization and propagation ofeach local cylinder while such interventions are not trivial toimplement to the user’s satisfaction in active shape or levelset type approaches.

Since direct interventions from the user is always possibleat each local cylinder, online corrections to the segmentationresult are possible. Future work includes an online segmen-tation algorithm that adaptively calculates edge values thattakes into consideration the local neighborhood. Interactivesegmentation of a fiber for the selected seed point will be in-corporated with the knowledge of the previously segmentedfibers. This iterative and interactive segmentation approachwill also enable easy segmentation of weaker fibers by seg-menting the strong edges first and eliminating the effects ofsegmented edges by incorporating them as inequality con-straints.

6. REFERENCES

[1] Wengang Zhou, Houqiang Li, and Xiaobo Zhou, “3d dendrite recon-struction and spine identification,” in Proc. of MICCAI ’08, Part II,Berlin, Heidelberg, 2008, pp. 18–26, Springer-Verlag.

[2] Delphine Nain, Anthony Yezzi, and Greg Turk, “Vessel segmentationusing a shape driven flow,” in in Proc. of MICCAI’04, 2004, pp. 51–59.

[3] Alexandra La Cruz, Matús Straka, Arnold Köchl, Milos Srámek, Meis-ter Eduard Gröller, and Dominik Fleischmann, “Non-linear model fit-ting to parameterize diseased blood vessels,” in IEEE Visualization2004, IEEE, Ed. Institute of Computer Graphics and Algorithms, Vi-enna University of Technology, 2004, pp. 400–393, IEEE.

[4] Mikal Rousson, Nikos Paragios, and Rachid Deriche, “Active shapemodels from a level set perspective,” 2003.

[5] Stanley Osher and James A. Sethian, “Fronts propagating with curva-ture dependent speed: Algorithms based on hamilton-jacobi formula-tions,” Journal of Computational Physics, vol. 79, pp. 12–49, 1988.

[6] Yoshinobu Sato, Shin Nakajima, Hideki Atsumi, Thomas Koller, GuidoGerig, Shigeyuki Yoshida, and Ron Kikinis, “3d multi-scale line filterfor segmentation and visualization of curvilinear structures in medicalimages,” in Proc. of CVRMed-MRCAS’97, London, UK, 1997, pp.213–222, Springer-Verlag.

[7] Alejandro F. Frangi, Ro F. Frangi, Wiro J. Niessen, Koen L. Vincken,and Max A. Viergever, “Multiscale vessel enhancement filtering,”1998, pp. 130–137, Springer-Verlag.

[8] Xin Gao Yisheng Zhu Stephen T.C. Wong Xiangjun Zhu, Zhong Xue,“Voles: Vascularity-oriented level set algorithm for pulmonary ves-sel segmentation in image guided intervention therapy,” in Proc. ofISBI’09.

[9] R. Manniesing, B. K. Velthuis, M. S. van Leeuwen, I. C. van der Schaaf,P. J. van Laar, and W. J. Niessen, “Level set based cerebral vasculaturesegmentation and diameter quantification in ct angiography,” MedicalImage Analysis, vol. 10, no. 2, pp. 200–214, April 2006.

[10] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and colorimages,” 1998, pp. 839–846.

[11] H.-C. Lee and D.R. Cok, “Detecting boundaries in a vector field,” IEEETrans. on Signal Processing, vol. 39, no. 5, pp. 1181–1194, May 1991.