Computer Vision and Image Understanding 117 (2013) 281–288

Contents lists available at SciVerse ScienceDirect

Computer Vision and Image Understanding

journal homepage: www.elsevier .com/ locate/cviu

Detecting end-effectors on 2.5D data using geometric deformable models:Application to human pose estimation q

Xavier Suau ⇑, Javier Ruiz-Hidalgo, Josep R. CasasUniversitat Politècnica de Catalunya, 1–3, Jordi Girona, 08034 Barcelona, Spain

a r t i c l e i n f o

Article history:Received 10 April 2012Accepted 23 November 2012Available online 6 December 2012

Keywords:Depth imageRange cameraEnd-effectorHuman pose estimationExtremities

1077-3142/$ - see front matter � 2012 Elsevier Inc. Ahttp://dx.doi.org/10.1016/j.cviu.2012.11.006

q This paper has been recommended for acceptance⇑ Corresponding author.

E-mail address: xavier.suau@upc.edu (X. Suau).

a b s t r a c t

End-effectors are usually related to the location of limbs, and their reliable detection enables robust bodytracking as well as accurate pose estimation. Recent innovation in depth cameras has re-stated the poseestimation problem. We focus on the information provided by these sensors, for which we borrow thename 2.5D data from the Graphics community. In this paper we propose a human pose estimation algo-rithm based on topological propagation. Geometric Deformable Models are used to carry out such prop-agation, implemented according to the Narrow Band Level Set approach. A variant of the latter method isproposed, including a density restriction which helps preserving the topological properties of the objectunder analysis. Principal end-effectors are extracted from a directed graph weighted with geodesic dis-tances, also providing a skeletal-like structure describing human pose. An evaluation against referencemethods is performed with promising results. The proposed solution allows a frame-wise end-effectordetection, with no temporal tracking involved, which may be generalized to the tracking of other objectsbeyond human body.

� 2012 Elsevier Inc. All rights reserved.

1. Introduction

Human pose estimation strategies are being widely applied tocountless applications. Knowledge on the position of differentbody parts, specially that of the head and limbs, is the key aspectof many interactive systems. Human pose estimation techniquesare helpful for a wide range of applications, from simple pointingand scrolling on a menu to complex gesture recognition. In orderto provide a truly immersive experience, marker-less body poseestimation is a must in this field [1,2].

Recent innovation in consumer oriented depth cameras has re-ceived growing interest in marker-less human pose estimation.Such cameras provide a pixel-wise depth estimation of the recordedscene in a working range, the most common ones operating in a typ-ical range between 0.5 and 5 m. Therefore, such new sensors providestraight-forward 3D information of the scene. Since the acquireddata is restricted to a single viewpoint, we denote it as 2.5D data.

Geometric deformable models (GDMs), proposed independentlyby Caselles et al. [3] and Malladi et al. [4], have proved perfor-mance and flexiblility at describing topology, as stated by Hanet al. [5]. GDM have been widely applied in the field of image [6]and volume [7] segmentation and component analysis.

Even if the GDM theory is formulated on the continuum, it maybe implemented in a discrete domain. An efficient and simple

ll rights reserved.

by J.K. Aggarwal.

implementation of the GDM is known as the Narrow Band Level Setmethod (NBLS), introduced by Adalsteinsson and Sethian [8], whichrestricts computation in thin bands surrounding a zero level. Peri-odic updates of these bands gradually cover the full area (or volume)of the analyzed data set, preserving its topology. The NBLS method isdefined for organized points in an evenly spaced grid (pixels in 2D,voxels in 3D), which limits accuracy due to resampling. Recently,Rosenthal et al. [9] have proposed an implementation of the NBLSmethod for unorganized 3D points, preserving their actual position.

In this paper, we propose an adapted version of the NBLS meth-od in the context of 2.5D data. The objective is to exploit connec-tivities over the depth surface to extract topological features.More precisely, the proposed method locates the end-effectors ofany 3D object. Since our work focuses on body pose estimation,the end-effectors are mainly the four extremities of a human body(Fig. 1). However, other 3D objects have been studied, with specialemphasis on the extraction of fingers of a human hand.

Human pose is inferred from the NBLS result and the obtainedend-effectors: firstly populating a graph from the previously com-puted NBLS and, secondly, extracting extremity pose with a short-est path algorithm from the end-effectors. The proposed method isevaluated against reference methods in Section 5.

1.1. Related work

Human pose estimation from 2.5D data is a current research to-pic as a result of the mentioned increasing performance of depthsensors. Shotton et al. presented in [10] a method to classify body

Fig. 1. Summary of the steps involved in the proposed algorithm for human pose estimation. From left to right: Foreground mask, R-NBLS propagation, R-NBLS filtering, end-effector graph nodes and end-effectors found with the associated skeleton.

C

282 X. Suau et al. / Computer Vision and Image Understanding 117 (2013) 281–288

parts using a Random Forests strategy. Body pose is then inferredfrom the body parts’ centroids. Baak et al. [11] combine local fea-ture matching with a database lookup, achieving a fast and robustend-effector tracking. Zhu et al. [12] propose a tracking algorithmwhich exploits temporal consistency to estimate the pose of a con-strained human model. Knoop et al.[13] propose a fitting of the2.5D data with a 3D model by means of ICP. Grest et al. [14] usea non-linear least squares estimation based on silhouette edges,which is able to track limbs in adverse background conditions.While these three methods focus on upper-body pose, Plagemannet al. [15] present a fast method which localizes body parts on 2.5Ddata at about 15 frames per second. Ganapathi et al. [16] extendthe work in [15] and extract full body pose by filtering the 2.5Ddata, using body parts’ locations.

Fig. 2. Empirical estimation of the law C for the Kinect camera, which gives theactual size of a pixel at a given depth level.

2. Preliminary concepts on 2.5D data

2.1. Input point cloud

2.5D data consists of a set of 3D points which corresponds to asampling of the scene surface from the camera viewpoint. The ob-ject of interest is usually segmented using depth cues. In this work,the set of 3D points corresponding to the object of interest is de-noted as the input point cloud D.

2.2. Apparent vs. physical area

Since 2.5D data is obtained from a single viewpoint, the conceptof apparent area arises. A pixel on the captured image correspondsto a physical surface Ap(z) which varies quadratically with the dis-tance z to the camera. Indeed, one may find a function CC relatingthe physical area of a given pixel to its apparent area, as a functionof z. We propose to find it empirically (Fig. 2) using a physical sur-face of known area (a DIN A2 sized surface at various distances).

Let S be a surface of unknown physical area AS and an apparentarea of NS pixels. We assume that S is sufficiently perpendicular tothe camera axis (we will see that this hypothesis is verified in thiswork), then AS � CC(zS) � NS is approximated as the physical area ofa pixel at the average depth level zS of S, times the number of pixelsin S. Knowing the physical area of a pixel avoids dealing with theapparent area, which is a great advantage of 2.5D sensors.

2.3. Connectivity

The raw data used in this paper consists of a point cloud ofunorganized 3D points. In order to find connected regions in thepoint cloud, a connectivity condition should be defined. We statethat a point p is (k, q)-connected if the number of points in a ball

of radius q centered at p is greater than k. Thus, a region will be(k, q)-connected if all its points are (k, q)-connected too.

3. Geodesic distance estimation using geometric deformablemodels and narrow band level sets

Geometric deformable models are based on the theory of curveevolution and the level set method [17]. The basic idea is to deforman initial curve or contour, which is registered to the data domain,depending on some pre-defined external and internal forces. Inter-nal forces will expand the actual curve over the data keeping itsmooth. External forces are computed from the available dataand have an effect on the curve evolution. By way of example,imagine a drop of corrosive acid eating an object. The initial curvewill be the drop at time zero, internal forces depend on the amountof acid in the drop, its corrosive power, etc. and external forces de-pend on the resistance of the material. Thus, corrosion will slowdown in resistive zones of the material and advance faster in areasmore prone to corrosion.

In our case, external forces are computed from the 2.5D dataD ¼ fxig 2 R3 and are defined to respect and preserve data featureslike topology or borders. We recall that D is the point cloud of theobject of interest.

Let /ðx; tÞ : R3 ! R be a level set function whose sole purpose isto provide an implicit representation of the evolving curve. Let alsoCðs; tÞ : R2 ! R3 be a contour parameterized by s as the zero levelset of /(x, t) and L0

t � D be the subset enclosed by Cðs; tÞ. Remarkthat C is parametrized in a two dimensional space, which is partic-ular to the 2.5D data case. Eq. (1) defines / at a given time instant t.In the level set notation, time represents the advance ofCðs; tÞ; t ¼ 0 being the time-stamp of the initial curve. In order to

Fig. 3. R-NBLS propagation example. The green points are the actual zero level setL0

t , and those with a thick black boundary form the actual contour Cðs; tÞ. The bluepoints in the middle are the candidate narrow band of with dL, with its contour alsomarked with thick point boundaries. Points labeled A (orange) are rejected becauseof the density condition in Eq. (2). Points labeled B (red) are rejected because of theproximity condition also in Eq. (2). (For interpretation of the references to color inthis figure legend, the reader is referred to the web version of this article.)

X. Suau et al. / Computer Vision and Image Understanding 117 (2013) 281–288 283

efficiently perform nearest neighbor queries to evaluate the Euclid-ean distance distE between points, D is organized as a kd-treestructure.

/ðx; tÞ ¼0 8x 2 L0

t

minfdistEðx; Cðs; tÞÞg 8x R L0t

(ð1Þ

The objective is to make the contour C evolve over D preservingthe topological properties of the latter. As cited in [5], the NarrowBand Level Set method is a simple solution to implement GDM evo-lution. An NBLS version for unorganized R3 points has also been pre-sented in [9]. In the NBLS method, the level set function / isevaluated in a thin layer surrounding the actual zero level set in or-der to update the zero level set for the next time instant. Such ap-proach limits the number of calculations to these few surroundingpoints.

In this paper, we propose to add a density condition to the exist-ing proximity condition. The role of this density condition is to fil-ter the data, especially those points near depth edges. Using theabove mentioned acid drop example, the density condition maybe considered as an external force, since it is implicitly derivedfrom the dataset. Propagation will slow down or stop in zones withlow data density, and continue in highly populated zones. Thus,only those end-effectors densely connected to the main body willbe considered, filtering sparsely represented or very thin ones.

We propose to update the zero level set according to Eq. (2). Wenote that time t may be considered as a discrete time, where tk: = -t + k with k 2 f0;Ng.

Ltþ1 ¼ fxig if/ðxi; tÞ < dL ðproximityÞ

and

xi is ðgL; dLÞ � connected ð densityÞ

8><>:

L0tþ1 ¼ L0

t [ Ltþ1

ð2Þ

The candidate narrow band (Lt+1) maximal width is dL, deter-mined by the proximity condition. The connectivity property of xi

is used as density condition, ensuring that the space surroundingxi is dense enough (at least gL are close enough to xi), as shownin Fig. 3. Therefore, dL and gL are parameters of the proposed NLBSvariant, called Restricted-NBLS or R-NBLS.

In order to complete the formulation, we should define how thecontour C is updated in the 2.5D context. In practice, the candidatepoints Lt+1 which are farther from the previous zero level set are ta-ken as the new contour Cðs; t þ 1Þ as shown in Eq. (3).

Cðs; t þ 1Þ ¼ fxsg 2 Ltþ1 with /ðxs; tÞ 234

dL; dL

� �ð3Þ

Thus, iterating through Eqs. (1)–(3) from an initial zero level setL0

t0¼0, the sufficiently dense zones of D will be covered.The geodesic distance between L0

t0and a given point xk which

was added to L0 at the time instant tk (or iteration k) may be calcu-lated with Eq. (4).

distG L0t0; xk

� �� k � dL þ /ðxk; tkÞ ð4Þ

The iterative R-NBLS method stops when the number of pointsNC

t of the actual contour Cðs; tÞ is smaller than a given stop thresh-old. The proposed iterative framework stops when NC

t ¼ 0 (see thesecond shape in Fig. 1 for an example).

3.1. Narrow band filtering by physical area

Zones of the scene being strongly oblique with respect to thecamera will be sparsely sampled with 3D points, and will not be ta-ken into account when constructing narrow bands. Consequently,the considered narrow band points are relatively parallel to the im-age plane axes, and the hypotheses in Section 2.2 are valid.

Narrow bands cover the visible and connected parts of the scenesurface. However, a band may be composed of points from botharms, since they contain points at similar distG (Eq. (4)). To separatepoints of a same band that belong to different contexts, narrowbands are filtered depending on their physical area. Indeed, a max-imal area Amax is set, so that any narrow band b with a physical areaAb larger than Amax is divided into a maximum of a regions,Nregions 6 a ¼ d Ab

Amaxe.Let N(b) be the number of points in b and �zb its mean depth. By

applying the approximation of Section 2.2, there exists a maximumnumber of points Nmax(zb) which keeps a constant at every depthlevel zb (Eq. (5)). Therefore, if �zb varies (i.e. a person moving to-wards or away from the camera), narrow bands will still be dividedinto no more than a regions.

Nmaxð�zbÞ ¼Amax

CCð�zbÞð5Þ

Remark that a standalone Amax restriction could result in very smallresidual regions as shown in Fig. 4. Therefore, besides the maximalarea condition given by Amax, some additional restrictions must beverified in the narrow band filtering step. More precisely, the fil-tered regions must be (gL, dL)-connected regions themselves (whichimplicitly forces a minimal region size of gL points). After the filter-ing step, a set of regions is obtained for every narrow band, all ofthem being (gL, dL)-connected. Some examples of the filtering stepare presented in Fig. 5.

4. Detecting end-effectors in a topologically weighted graph

End-effectors are topologically prominent protuberances in theobject under study, restricted to the viewpoint of the range cam-era. A method to detect end-effectors from the result of the R-NBLSmethod is discussed in this Section (see Fig. 1).

4.1. Graph root

The proposed R-NBLS method requires the specification of aninitial zero level set L0

t0¼0 as starting point. Such origin region,called graph root, may be a single 3D point or a set of points. The

Fig. 4. On the left, a narrow band filtering with a standalone Amax restriction, which results in a Nmax = 10 maximal size. Note the residual region of five points (in orange). Onthe right, the filtering step with the additional restriction on connectivity with gL = 5. The non-filled points have been filtered since they are not (gL, dL)-connected.

Fig. 5. Three examples of narrow band filtering. The obtained regions are randomlypainted. In this case, an Amax = 70 cm2 has been applied together with a (gL = 30,dL = 4 cm)-connectivity.

284 X. Suau et al. / Computer Vision and Image Understanding 117 (2013) 281–288

definition of the graph root will strongly depend on the applica-tion. In this paper, the cases of a whole human body and a humanhand are studied, with their specific graph roots.

In the human body case, we propose to use a straight line asgraph root (blue1 line in Fig. 1). Such line connects the centroidxC of D with xM, the latter being the midpoint between xC andthe head position xH, which is obtained with [18]. Those pointsplaced at a given distance d0 of lC are labeled as initial zero levelset L0

t0, from which the R-NBLS propagation can start. Despite head

estimation, which exploits some temporal information to increasetracking robustness [18], the rest of the proposed algorithm isframe-wise, without any temporal dependency.

4.2. End-effector graph construction

In general, R-NBLS filtered regions belong to prominent parts ofthe analyzed object (i.e. arms, legs) due to the band splitting con-sidering connectivity (Section 3.1). The objective of this paperbeing that of finding end-effectors, it seems reasonable to use thesecontext-wise regions.

In a consecutive phase to the narrow band filtering (Section3.1), a graph is constructed on the filtered regions. The centroidof every region is taken as a graph node, with an associated crea-

1 For interpretation of color in Fig. 1, the reader is referred to the web version ofthis article.

tion time tk coming from the R-NBLS propagation step explainedin Section 3.

Graph nodes are linked in pairs with graph edges. We proposeto only include those edges which link a source node ni with timeti and a sink node nj with time tj such that i < j, resulting in a direc-ted graph (from inner to outer narrow bands). This way, any pathconstructed on the graph will be consistent with the node creationinstants, not linking nodes with previously created ones.

A distance weight wi,j is calculated for every edge linking nodesni and nj, with an additional distance penalty depending on thetime elapsed between the creation of the nodes. Such penalty lim-its the construction of graph paths with strong jumps in creationtime, this effect happening only in strictly necessary occasions. A10% gain is added to the penalty a = 1.1, so that it penalizes slightlymore than an integer number of jumps. The proposed nodeweights are calculated with Eq. (6).

wi;j ¼ jni � njj|fflfflfflffl{zfflfflfflffl}distance

þðj� i� 1Þ � dL � a|fflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflffl}penalty

ð6Þ

4.3. End-effector estimation: shortest path from farthest level

A Dijkstra shortest path algorithm is run on the graph con-structed in Section 4.2. End-effectors are extracted as the shortestpaths from the farthest nodes to the graph root. Paths are searchedstarting at the node with greater tk and ending at xM (arms) or xC

(legs). If it does not exist any path from that node, successive nodesare taken as path sources by decreasing tk until all paths have beenfound. Indeed, two paths ending at xM are searched and labeled asarms, and two other paths are searched to end at xC, which aretaken as legs. Since our work focuses on human body, end-effectorsare often referred as extremities. Some conditions should beverified in order to accept a path as and end-effector of a humanbody.

� A path must have at least three segments, avoiding too shortnoisy detections.� For a given graph, those nodes which belong to an already

accepted path become unaccessible for further pathestimations.� Those nodes at a geodesic distance smaller than 30 cm from the

central line lC are not taken into account, since we are lookingfor human extremities. Such restriction limits the detection ofextremities starting close to the body centroid. We assume thisdraw-back of the algorithm, as shown in Fig. 8.

When the arms and legs have been found, path search stops.The result computed by the presented algorithm constitutes the

X. Suau et al. / Computer Vision and Image Understanding 117 (2013) 281–288 285

end-effector positions along with a skeletal-like structure describ-ing the limbs. It should be noted that some poses may result inundetectable extremities, since their topological prominence isnot clear enough. Therefore, only those end-effectors which aresufficiently detached from the body will be detected. The headhas been found as in [18]. Nevertheless, note that it could be de-tected as an extra path from xM.

4.4. Right and Left extremity decision

Taking advantage of the extremity graph, a decision whether alimb corresponds to the right or left hand is taken. No temporalcues are involved in the decision.

For hands, the direction of first segment (t0 to t1) of every graphpath (gA and gB) is calculated, obtaining two vectors fA and fB. Asimple decision depending on the orientation of fA and fB is per-formed, taking as right hand the path with fi more oriented tothe horizontal axis to the right (positive X coordinates). Remarkthat using the first graph segment is strongly invariant to the posi-tion of the end-effector associated to the graph path. A similar rea-soningh is done for feet, using their two graph paths. Yet being abasic classification approach, experimental results show that theproposed decision framework is effective (Section 5.3).

5. Experimental results

The following results have been obtained with a Kinect sensorwhich delivers both depth and color images at a frame-rate ofabout 25 fps with a resolution of 640 � 480 pixels. The color infor-mation is discarded and only the depth estimation is exploited inthe experiments below. In Section 5.3, the proposed method iscompared to two reference methods [10,16].

5.1. Effect of the parameterization on the human pose estimation

Only dL and Amax are important parameters, gL being a filteringparameter which may be kept invariant for a given sensor. Forthe Kinect camera, a value of 0.5 points per cm2 during propagationhas proven to be adequate. Therefore, the value of gL is tightly re-lated to the dL parameter (gL = 0.5 � p � dL

2).The narrow band maximal width, dL, determines the resolution

of the propagation, and also the areas which will be covered. A toolow value of dL leads to propagation cuts, the advancing contour

Fig. 6. (a) Low values of dL provide more precision for the detection of topological prommaximal area parameter (left to right, Amax = 20 cm2, 70 cm2 and 200 cm2) determines theof close legs are considered as trade-off values. Indeed, the narrow bands covering the lexample, 70 cm2 seems to be a convenient trade-off.

being too poor (few or no points) at some topologically narrowzones. On the other hand, a high dL value affects precision andextremities are less frequently detected (greater topological prom-inence is needed). Fig. 6a shows two R-NBLS propagations withdL = 4 cm and dL = 10 cm. Both arms are close to the body and theirtopological prominence cannot be detected with a large dL.

The maximal area Amax controls the population of the end-effec-tor graph. The smaller Amax, the more nodes are included in thegraph, allowing more freedom for finding plausible paths. How-ever, extremity detection is less stable with very low Amax values.On the contrary, if Amax is increased, the graph is poorly populated,resulting in a very rigid extremity estimation. In Fig. 6b, threeexamples are shown to illustrate the effect of various Amax valueson the extremity detection.

5.2. Effect of the parameterization on the detection error and detectionrate

In order to evaluate the proposed method, more than 1000depth images have been manually marked (hands and feet) asground-truth. Only those extremities noticeable to the naked eyeon the depth images are marked, avoiding guessing limbs’ positionfrom the input data.

Results after various parameterizations are summarized inFig. 7. The error is measured as the Euclidean distance betweenthe 3D ground-truth points and the estimated ones. In order to in-clude the variance of the estimation, we plot �e and re on the hor-izontal axis, where �e is the average error and re the standarddeviation. On the vertical axis, the percentage of detections overthe number of ground-truth detections is plotted, taking into ac-count only those detections with an error smaller than 30 cm.Therefore, parameterizations with higher detection rate and lowerð�eþ reÞ will provide the best results.

Experimental results show that parameterization B = (dL = 8 cm,Amax = 70 cm2) obtains the best results with the Kinect camera.More precisely, it achieves extremity detection with an error smal-ler than 30 cm (maximal size of a foot or hand) in about 77% of theover one thousand annotated frames. The average error is�e ¼ 3:94 cm and its standard deviation re = 4.79 cm. The percent-age of detection increases while the error does so. Such effectshows the trade-off between obtaining many poor detections orless precise detections.

A summary of different situations has been presented in Fig. 8to show how the proposed human pose estimation algorithm

inence, even if the resulting paths are less straight (noticeable at the legs). (b) Thepopulation of the end-effector graph. Values of Amax which allow a proper detection

egs will split into two regions, allowing the computation of two paths to xC. In the

Fig. 7. Detection error vs. detection rate (%) for various parameterizations (dL, Amax).A = (10 cm, 120 cm2), B = (8 cm, 70 cm2), C = (6 cm, 60 cm2), D = (5 cm, 50 cm2) andE = (4 cm, 40 cm2). Parameterization B seems to provide the best trade-off, with anaverage error of about 3.94 cm and a standard deviation of 4.79 cm for a detectionrate of about 70%.

Fig. 9. Average precision comparison with [10,16] over the 27 sequences providedby [16].

286 X. Suau et al. / Computer Vision and Image Understanding 117 (2013) 281–288

performs with various human poses. Both easy situations (crosspose, farther left) and more difficult ones (punching, farther right)are properly solved, providing a 3D estimation of the position ofthe extremities. When extremities are not topologically prominent(i.e. third pose from the left), they are not detected. This is a logicaldraw-back of the proposed method. These estimations are ob-tained without any temporal tracking of the extremities, providinga frame-wise solution to the human pose estimation problem.

5.3. Evaluation against reference methods

Two reference methods are used to evaluate the proposedmethod. In [10], Shotton et al. propose a body part classificationby means of a Random Forest strategy. Ganapathi et al. proposein [16] a model-based approach exploiting temporal consistency.They also provide a dataset consisting of 27 sequences of increas-ing difficulty, recorded with a Time-of-Flight (TOF) camera. More-over, ground-truth positions obtained with a motion capturedevice are provided. The experiments presented hereafter are ob-tained on the dataset in [16].

5.3.1. Classification precisionThe proposed method detects head, hands and feet of a human

body. Therefore, we select the subset of markers in the dataset of[16] that represent these body parts. In Fig. 9, a summary of the ob-tained average precision (AP) is provided. The proposed methodoutperforms [16], only being slightly surpassed in the cases ofthe head and right foot. The method in [10] obtains slightly betterresults in average. However, it is a specific classification method,

Fig. 8. Some examples of the proposed

whilst the other two methods are focused on detection, makingno classification effort.

Regarding the average detection error, we compute the 3D errorbetween the obtained end-effectors and the selected ground-truthmarkers. Results are presented in Fig. 10, compared to the resultsobtained by [16]. The proposed method behaves in a similar man-ner over the whole dataset, obtaining an average error of about9 cm even in the most challenging sequences (24–27). The methodin [16], obtains a slightly better detection error in the first se-quences, strongly degrading its results when facing the challengingsequences.

As far as processing speed is concerned, the proposed methodexecutes at about 57 fps using the dataset in [16] (176 � 144 reso-lution). In this paper, we have used a single core of an Intel XeonCPU at 3 GHz. The work in [16] achieves a frame rate of about4 � 10 fps with a specific GPU implementation. The method in[10] claims a 50 fps execution frame-rate on full Kinect images(640 � 480) using an 8-core desktop CPU, and 200 fps using a ded-icated powerful GPU. In [11], a frame-rate of 60 fps is achievedusing a commercial CPU and a similar resolution.

5.4. Application to finger detection

The proposed algorithm to detect end-effectors may be appliedto other objects besides the already presented human body case.With a suitable zero level set L0

t0and an adapted parameterization

(dL, Amax), the prominent end-effectors of any object may be found.We obtain hand positions as in [18], onto which we apply the

proposed R-NBLS end-effector detector, aiming to detect the num-ber of extended fingers. Some finger detection examples are shownin Fig. 11, where one, two, three and four fingers are detected. Inthis example, the person is located about 2.5 m far from the rangecamera. The initial zero level set L0

t0is set as the centroid of the

hand blob (graph root), and the parameterization (dL, Amax) =(2 cm, 7 cm2).

human pose estimation algorithm.

Fig. 10. Detection 3D error comparison with [16] over the 27 sequences provided by [16].

Fig. 11. Application to finger detection. Hands have been obtained from the work in [18] (first row).

X. Suau et al. / Computer Vision and Image Understanding 117 (2013) 281–288 287

6. Conclusion

We propose an end-effector detection algorithm based on topo-logical propagation over 2.5D data. In order to carry out such prop-agation, Geometric Deformable Models are used. An extension ofthe Narrow Band Level Set formulation, named R-NBLS, is proposedto implement the GDM, adding a density restriction to better re-spect the topological properties of the analyzed object. The ob-tained level set provides a fast method to calculate geodesicdistances over the original 2.5D data.

A skeletal-like structure of the object under analysis is esti-mated independently at each frame, without any temporal track-ing of the extremities. In the specific human body case, suchskeleton is tightly related to human pose. We provide a simplemodel to initialize the R-NBLS method in the case of human body.

The proposed method performs about 5� to 50� faster than[16], even in the adverse case of comparing our CPU implementa-tion to the GPU implementation in the reference work. Our pro-posal is also faster than [10], even if they use eight CPU coresinstead of the single core used in our proposal. Using a similar res-olution, a frame-rate of 60 fps is achieved by [11], taking advantageof using a pre-computed training dataset.

R-NBLS outperforms the method in [16] in classification preci-sion, and achieves similar results in terms of detection error. Themethod in [10] obtains a slightly better classification precision,taking advantage of a large training dataset and a dedicated classi-fication task.

Other objects have been studied besides human body, with spe-cial interest in finger detection given a hand object. Fingers are de-tected at low resolutions, with a person placed about 2.5 m farfrom the range camera.

Including topological borders or frontiers is one of the mainforeseen points for further work. Such borders could help to im-prove the propagation in order to better respect the topology.We envisage to use color information and other local descriptorsto complement the depth estimation, which will help solvingambiguities, increasing the detection rate.

Acknowledgments

The research leading to these results has received funding fromthe European Union’s Seventh Framework Programme (FP7/2007–2013) under Grant Agreement No. 248138.

This work has been partially supported by the Spanish Ministe-rio de Ciencia e Innovación, under Project TEC2010-18094

Appendix A. Supplementary material

Supplementary data associated with this article can be found, inthe online version, at http://dx.doi.org/10.1016/j.cviu.2012.11.006.

References

[1] T.B. Moeslund, A. Hilton, V. Krüger, A survey of advances in vision-basedhuman motion capture and analysis, Computer Vision and ImageUnderstanding 104 (2-3) (2006) 90–126.

[2] R. Poppe, Vision-based human motion analysis: an overview, Computer Visionand Image Understanding 108 (2007) 4–18.

[3] V. Caselles, F. Catté, T. Coll, F. Dibos, A geometric model for active contours inimage processing, Numerische Mathematik 66 (1) (1993) 1–31.

[4] R. Malladi, J.A. Sethian, B.C. Vemuri, Shape modeling with front propagation: alevel set approach, TPAMI 17 (2) (1995) 158–175.

[5] X. Han, C. Xu, J. Prince, A topology preserving level set method for geometricdeformable models, TPAMI 25 (6) (2003) 755–768.

288 X. Suau et al. / Computer Vision and Image Understanding 117 (2013) 281–288

[6] M. Maška, P. Matula, A fast level set-like algorithm with topology preservingconstraint, in: International Conference on Computer Analysis of Images andPatterns, Springer, 2009, pp. 930–938.

[7] R. Whitaker, D. Breen, K. Museth, N. Soni, A framework for level setsegmentation of volume datasets, in: International Workshop on VolumeGraphics, Vol. D, 2001, pp. 159–68.

[8] D. Adalsteinsson, J. Sethian, A fast level set method for propagating interfaces,Journal of Computational Physics 118 (2) (1995) 269–277.

[9] P. Rosenthal, V. Molchanov, L. Linsen, A narrow band level set method forsurface extraction from unstructured point-based volume data, in:International Conference on Computer Graphics Visualization and ComputerVision, 2010, pp. 73–80.

[10] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman,A. Blake, Real-time human pose recognition in parts from single depth images,in: CVPR, 2011, pp. 1297–1304.

[11] A. Baak, M. Meinard, G. Bharaj, H.-p. Seidel, C. Theobalt, M.P.I. Informatik, Adata-driven approach for real-time full body pose reconstruction from a depthcamera, in: ICCV, 2011.

[12] Y. Zhu, B. Dariush, K. Fujimura, Controlled human pose estimation from depthimage streams, in: ICCV Workshops, IEEE, 2008, pp. 1–8.

[13] S. Knoop, S. Vacek, R. Dillmann, Sensor fusion for 3D human body trackingwith an articulated 3D body model, in: ICRA, IEEE, 2006, pp. 1686–1691.

[14] D. Grest, V. Krüger, R. Koch, Single view motion tracking by depth andsilhouette information, Lecture Notes in Computer Science 4522 (2007) 719–729.

[15] C. Plagemann, V. Ganapathi, D. Koller, S. Thrun, Real-time identification andlocalization of body parts from depth images, in: ICRA, IEEE, 2010, pp. 3108–3113.

[16] V. Ganapathi, C. Plagemann, D. Koller, S. Thrun, Real time motion capture usinga single time-of-flight camera, in: CVPR, 2010, pp. 755–762.

[17] J.A. Sethian, Level set methods and fast marching methods, CambridgeMonograph on Applied and Computational Mathematics, vol. 39, CambridgeUniversity Press, 1999.

[18] X. Suau, J. Ruiz-Hidalgo, J.R. Casas, Real-time head and hand tracking based on2.5D. data, IEEE Transactions on Multimedia 14 (2012) 575–585.