A Hierarchical Shape Tree for Shape Classification

Y. Li1, J. Zhu1,F. L Li1

1Wuhan University,China.Email: liying.yunran@gmail.com

Abstract

This paper presents a novel approach to hierarchical shape classification.We combine two shape features:

contour and skeleton. Weights of two features are learned through large-margin optimization. The

proposed approach uses a shape tree to efficiently represent the similarity of different shape classes. The

tree is generated offline by a bottom-up clustering approach using stochastic optimization.A coarse-to-fine

matching strategy is adopted to have a high classification accuracy.Bayesian classifier is used to perform

the final decision.The proposed method was tested in a variety of challenging shape datasets.The results

show great improvement over many previous algorithms.

Keywords: Shape Classification, Hierarchical Model, coarse-to-fine matching, Bayesian classifier

1 Introduction

An important goal in image analysis is to classifyand recognize objects. As a significant factor ofobjects, shape is an important research directionin computer vision, so to study the underlying sta-tistics of shape population and classify shapes ba-sed on the statistics information have been heavilyresearched recently. In this paper ,we focus onclassifying 2D shapes of closed contour.

A key issue in shape analysis is finding a goodrepresentation. Smart features for shape matching,such as shape context[1], have been greatly studiedrecently. This line of the work [1, 2, 3, 4] has itsown limits,though they have produced increasin-gly promising results on matching.Matching-basedalgorithms have difficulty capturing the large intra-class variation on few training samples,while it istime-consuming to perform matching one-to-oneon large scale data[5]. On the other hand, twoconcepts have been widely explored in shape clas-sification, contour and skeleton.Contour-based ap-proaches [6, 2, 3, 4] are often good at represen-ting detailed shape information and easy to ex-tract,but sensitivity to articulation and non-rigiddeformation[7] limits its performance.Skeleton-basedmethods[8, 9, 10, 11] can cope well with non-rigiddeformation, but difficult to extract and carry roughstructural information only.However, two conceptsseem to be isolated by researchers. Actually, twodescriptors are complementary to each other andcan be combined to obtain a significant improve-ment on shape classfication[5]. In our paper, weuse both contour and skeleton as shape represen-tation.

However, due to presence of outliers and noise,contour and skeleton are still not enough to capturethe intra-class variants. So sometimes the accuracyof shape classification is surprisingly low.Motivatedby the idea of [12, 13, 14, 4], this paper propo-sed a hierarchical model of shape classification. Acoarse-to-fine strategy of matching has been adop-ted to improve the performance of classification.Our system is robust to shape noise and outliers.

The main contributions of this paper lie in: (1)Two shape descriptors,contour and skeleton, havebeen efficiently combined by large-margin optimi-zation and well fit to the framework of Bayesianclassifier;(2) A hierarchial shape tree was learnedand constructed offline and a coarse-to-fine mat-ching strategy was very efficient to capture thestructural information. The outline of the paperis as follows: Section 2 discusses shape represen-tation. Section 3 provides a detailed discussionabout our hierarchial models including similaritymeasures and hierarchical matching method. Theexperiments and results listed in section 4 show theperformance of our shape model.Conclusion andfuture work has been put in section 5.

2 Shape Representation

This section describes the representation of ourshape model.Usually, contour and skeleton are twocommon descriptors of shape. [5] integrated bothof contour and skeleton by a Bayesian classifier andachieves a very high accuracy in a large database.In this paper ,we use the same representation as[5]. Below is a brief description of our shape repre-

978-1-4244-9630-3/10/$26.00 ©2010 IEEE

sentation.

2.1 Contour Segment

In [6],contour segments were extracted based oncurvatures and were used to shape classification.Instead of curvatures, this paper extracts them byDiscrete Curve Evolution(DCE)[15].Let S denotesthe shape. Firstly, we apply DCE method to ob-tain a simplified polygon on S with the verticesdenoted as

−→u = (u1, u2, . . . , uN), (1)

where N is the number of vertices.The contourbetween two vertices is extracted as a contour seg-ment.Hence the set of contour segment can be de-noted as below:

S = {cij = (ui, uj), i 6= j, i, j ∈ 1, . . . , N}. (2)

Each segment c is sampled with n equal distancepoints {x1, x2, . . . , xn}. Then c is transformed to avector −→v ,which we call signature[6]. The transfor-mation is done by mapping x1 to x′1 = (0, 0) andxn to x′n = (1, 0), which allows us to compute theplanar similarity transform mapping the remainingpoints to x′2, . . . , x′n−1 in the normalized referenceframe. Each signature v is then represented by avector v = (x1, y1, . . . , xn, yn)

T. Thus we definethe set of all signatures as

V{S} = {cij = (−→v 1, . . . ,−→v |S|}. (3)

where |S| is the total number of contour signatures.Fig.1 shows an example of contour segment.

Figure 1: (a) is a contour of a leopard. (b) shows the

critical points on the contours, and the two contour

segments in red.(c) is the corresponding signature of

two contour segments in red.

2.2 Skeleton Path

Skeleton-based shape matching and recognition al-gorithms have been heavily studied[8]. Here wealso use DSE method [15] to extract skeleton. Aspointed out as [15], skeleton paths by DSE are verysimple but informative. This is because the insta-bility of the junction points of skeletons is avoidedin skeleton path and different shape instance forthe same category usually have very similar radiisequence. Let K(S) be the skeleton of S, and

e(S) = {ei, i = 1, . . . ,M} be all the endpoints ofK(S) with M as the total number of endpoints.The set of skeleton path can be expressed as

H{S} = {hij = (ei, ej), i 6= j, i, j ∈ 1, . . . , N}.(4)

Fig.2 shows some example skeleton paths. For eachskeleton path h, we sample n number of equaldistance points. At each skeleton points si, theradius r(si) is approximated bye the value of dis-tance transform DT (si). So a vector of the radiiof the maximal disks of a skeleton path is denotedas r = (r(si), . . . , r(sM )). Also,to make each rinvariant to the scale,we normalize it by

−→r =DT (s)

1M

∑DT (si)

. (5)

Therefore, we obtain a set of radus vectors:

R{S} = {−→r 1, . . . ,−→r |HS|}. (6)

Figure 2: (a) is a skeleton of a leopard, and the red

points e1,e2,e3,e4are endpoints computed by DCE. (b)

and (c) shows skeleton paths.

3 A Hierarchical Model

Recent works have put a lot of attention on hie-rarchical models[12, 14, 4], and these models showgreater performance than some other techniques.Hereby, we present a hierarchical model to shapeclassification. The idea is quite simple and motiva-ted by efficiency consideration. Usual classificationmethods compute the similarity measure betweenthe query shape and all the shape class in trai-ning set. However, due to the presence of outliers,some quite naive errors happen such as classifyinga horse into a pen class. In this paper, we show ahierarchical model which can significantly decreasethe possibility of such errors and efficiently improvethe accuracy rate.

3.0.1 Distance Measure

The feature set of a shape is {V(S),R(S)}. For aset of training shapes of the same class SS, twosets are collected:

V(S) =

|SS|⋃

i=1

V(S),R(S) =

|SS|⋃

i=1

R(S), (7)

where |SS| is the number of training shapes. Thedistance of two contour segments −→v i and −→v j isdefined as:

D(−→v i,−→v j) =

n∑

t=1

(−→v i(t)−−→v j(t))2

−→v i(t) +−→v j(t). (8)

Similarly, the distance between two skeleton paths−→v i and −→v j is :

D(−→r i,−→r j) =n∑

t=1

(−→r i(t)−−→r j(t))2

−→r i(t) +−→r j(t). (9)

The basic idea for achieving an efficient shape re-presentation is to group similar shape classes toge-ther with a distance parameter which needs to cap-ture the dissimilarity between the different shapeclasses. First, we give the definition of shape-to-class distance. As pointed out in[16], simple KNNclassification method has higher accuracy. Here,assume the shape-to-shape distance between a shapeS and another shape S′ is

D(S → S′) = w1 ·1

|V(S)|

∑D(v,NN(v))+

w2 ·1

|R(S)|

∑D(r,NN(r)),

v ∈ V(S), r ∈ R(S).(10)

where NN(v) denotes the K nearest neighbors ofv in V(S′), similar is NN(r), and w = (w1, w2)is the weight of contour and skeleton path res-pectively.Note that this definition of distance isasymmetric. For each shape ,we can learn a weightvector using the framework in [17]. In the weightlearning stage, supposing I is a shape of categoryC, we find a pair of J and K such that J is ashape of the same category C and K is a shape of adifferent category. The learning algorithm enforcesthe following condition:

D(I → K) > D(I → J) (11)

⇒ (wI, dIK) > (wI, dIJ) (12)

⇒ (wI, xIJK) > 0, (13)

where xIJK = dIK − dIJ, and dIK is the dis-tance vector between shape I and K, as defined in10.Supposing T such pairs for I from the trainingset is constructed, thus large-margin optimizationis formulated as follows:

min1

2wTw + C

T∑

i=1

ζi (14)

s.t. : wTxi ≥ (1 − ζi), ζi ≥ 0, ∀i = 1, . . . , T (15)

w � 0. (16)

Therefore the distance between two shape class Ciand Cjis defined as

D(Ci, Cj) = (∑

St∈Ci

∑

Sk∈Cj

(D(St → Sk)+D(Sk → St)))/2.

(17)Therefore, we can compute the distance matrix ofN shape classes.

3.1 Construction of Shape Tree

Given all the training examples, we can constructa hierarchical shape tree in which a node representa shape class. We use the clustering method in[18] to cluster all the shape class according to thedistance matrix of N shape classes. In each cycle,we just let the number of cluster centers K be 2.When this strategy is applied recursively, this grou-ping leads to a binary shape tree, see Fig.3. Thestarting point is a set of shape templates from allthe training set which cover inherent object shapevariations and allowable geometrical transforma-tions. A shape tree is subsequently constructedon top of these shape templates. The proposedalgorithm involves a bottom-up approach and apartitioned clustering step at each level of the tree.The stop criteria is that in the final level a nodecan contain at most two kinds of shape classes.SeeFig.3 for a typical partial view. Observe how theshape similarity increases toward the leaf level.

Figure 3: Example for the construction process of a

hierarchical shape tree.

3.2 Hierarchical Matching

Online matching can be seen as traversing the treestructure of shape templates.Given a query shape,processing a node involves the computation of shape-to-node distance which is defined as the mean valueof all shape-to-class distances of the query shapeto the shape classes in the node.For the locationwhich has the minimum shape-to-node distance ,thecorresponding child nodes are added to the list ofnodes to be processed.For the last but one level,search does not propagate to the subtree. This isbecause a finer bayesian classifier will be applied

to the last level of nodes. The above coarse-to-fine approach can greatly improve the accuracy ofbayesian classifier. To apply a bayesian classifierdirectly to all the shape classes can always producestupid mistake such as predicting a dog as a bone.However, by hierarchically matching a query shapeto shape tree, this naive mistake can be avoided.

3.3 Bayesian Classifier

We use a Bayesian classifier to perform the finestclassification after the hierarchical matching. Incontrast with [19], which also use Bayesian clas-sifier with contour features, our system computeshape-to-class distance rather than the distance toindividual objects, and we don’t need the compu-tation of correspondence of features. Given aninput shape S, posterior probabilities of classesgiven contour segments and skeleton pathψ{S} =V{S}

⋃V{S} is computed by Bayes’ rule:

P (ci|ψ{S}) =p(ψ{S}|ci)P (ci)

p(ψ{S})(18)

where mathrmci is one of the C classes.The class-conditional probability for observing ψ{S}given thatS belongs to class ⌋i is :

p(ψ{S}|ci) = Σp(ψ{S}|ψ{ci})P (S|ci). (19)

Hereby for any descriptors including contour seg-ments and skeleton paths,we assume an isotropicK-dimensional normal density around each coeffi-cient vector in the dataset, with scale factor σ:

p(ψ{S}|ψ{ci}) =∑ 1

(2πσ2)K/2 exp(−D2(S,S′)

2σ2 ),

∀S′ ∈ ψ{ci}(20)

The final decision is as below:

c∗ = argmaxP (ci|ψ{S}) (21)

4 Experiments

We test our algorithm on three popular shape data-bases [10, 3, 5]. All the experiments are conductedusing Matlab on a PC with 1.5 GHZ CPU and512M RAM. The results demonstrate that our hie-rarchical model can improve the performance ofshape classifier.

4.1 Kimia dataset

We apply our method to Kimia dataset [10] asshown in Fig.4, which includes 18 classes, and eachclass is consisted of 12 shapes. The result is quitepromising with only 4 error in all. Fig.5 showsall the errors. From Fig.5 we can see that ourmethod can improve the accuracy because it avoidssome simple mistakes due to occlusion and outliers

Figure 4: Example shapes for the kimia shape dataset.

Method Bai’s[20] Sun’s[6] OursAccuracy 94.4% 97.2% 98.8%

Table 1: Results on Kimia dataset[10]. Ours is better.

in presentation. Averagely, a classification taskcan take only 0.5 minute.The parameter in theexperiment is set as C = 1.2, n = 100, σ = 1.5.To have a better illustration of our method, wecompare our method to two previous work[20, 6].In [6],only contour segments are used while in [20]only skeleton paths are used. Both [20, 6] haveapplied Bayesian classifier. As Tabel1 shows thatour method outperforms the other two algorithms.This is due to the fact that contour segments andskeleton paths are complementary and our hierar-chical model can well capture the intra-class varia-tions.

Figure 5: All the errors in the experiment on the kimia

dataset.(a) shows that a hammer is misclassified as a

bone.(b) shows a classic car is misclassified as a brick.

(c) shows a brick is misclassified as a car.

4.2 MEPG-7 dataset

Figure 6: Examples of Mepg-7 dataset, one shape per

class.

We also test the proposed approach on a famousshape dataset MEPG-7 [3]which contains 70 classesof shapes, with 20 examples per class, for a totalof 1400 shapes. Fig.6 shows a few example shapesfrom this dataset. There are significant intra-classvariation including, e.g., variation due to geometricdifferences, structural difference, and articulation.To have a fair comparison, we use the same ex-perimental setup as [5]. Half the shape dataset isused as training samples. In [5], contour segmentsand skeleton paths are combined by a simple way.By contrast, our method learns the weights of bothdescriptors and applies a coarse-to-fine method bymatching a hierarchical shape tree. This process

Method Bai’s[5] Sun’s[6] OursAccuracy 96.6% 91.1% 97.8%

Table 2: Results on Mpeg-7 dataset[3]. Our hierarchi-

cal model works better.

can give a rise to the results of classification. SeeTable2. Averagely, a classification task can takeonly 0.5 minute.The parameter in the experimentis set as C = 1.1, n = 100, σ = 1.3.We evaluatedthe robustness of the method to the parameter va-lues. The values of n and C do not affect accuracyas long as the resolution is high enough to capturethe shape features. We test the sensitivity of σ,and found the accuracy is stable over most of thatrange(See Fig.7).

Figure 7: Classification vs. σ.

4.3 Animals dataset

Figure 8: Examples of Animals dataset, two shapes

per class.

Recently, a new dataset [5] which consists of 20classes with 100 shapes per class. These shapeshave much larger intra-class variations and inter-class similarities than the MPEG-7 dataset. Alsothere is severe self-occlusion in the dataset. Fig.8shows some examples in the dataset.The same ex-perimental setup as section4.2, we compare theproposed method with some famous algorithms onshape classification. The results are shown in Table3.

5 Conclusion

This paper presented a shape classification methodbased on a hierarchical model.Contour segments

Method IDSC[2] CS[6] CS&SP[5] Ours

Accuracy 73.6% 71.7% 78.4% 80.0%

Table 3: Results on Animals dataset[5]. Our hierarchi-

cal model improves the accuracy of shape classification.

and skeleton path are combined for shape repre-sentation which can well capture the variation ofshapes. The weight of two features are learnedby large-margin optimization.A hierarchical shapetree is constructed using a partitioned clusteringmethod. We apply a coarse-to-fine method whichfirst match shape to nodes of shape tree by KNNshape-to-class distance, and then a bayesian classi-fier to used to a more accurate task of classification.The empirical evaluation on several popular butvery challenging datasets shows great improvementof previous works.

References

[1] J. M. S. Belongie and J. Puzicha, “Shapematching and object recognition using shapecontexts,” PAMI, vol. 24, no. 4, pp. 509–522,2002.

[2] H. Ling and D. Jacobs, “Shape classificationusing the inner-distance,” PAMI, vol. 29,no. 2, pp. 286–299, 2007.

[3] L. Latecki and R. Lakamper, “Shape simi-larity measure based on correspondence ofvisual parts,” PAMI, vol. 22, no. 10, pp. 1185–1190, 2000.

[4] P.F.Felzenszwalb and J. D. Schwartz, “Hie-rarchical matching of deformable shapes,” inProc. CVPR, 2007.

[5] W.-Y. L. X. Bai and Z. Tu, “Intergratingcontour and skeleton for shape classification,”in Workshop on Proc. ICCV, 2009.

[6] K. Sun and B. Super, “Classification ofcontour shapes using contour segment sets,”in Proc. CVPR, 2005.

[7] A. B. A.M. Bronstein, M.M. Bronstein andR. Kimmel, “Analysis of two-dimensionalnon-rigid shapes,” IJCV, vol. 78, pp. 67–88,2008.

[8] X. Bai and L. Latecki, “Path similarity ske-leton graph matching,” PAMI, vol. 30, no. 7,pp. 1282–1292, 2007.

[9] E. E. C. Aslan, A. Erdem and S. Tari,“Disconnected skeleton:shape at its absolutescale,” PAMI, 2008.

[10] P. K. T.B. Sebastian and B. Kimia, “Recogni-tion of shapes by editing their shock graphs,”PAMI, vol. 26, no. 5, pp. 550–571, 2004.

[11] S. D. K. Siddiqi, A. Shokoufandeh and S. Zu-cker, “Shock graphs and shape matching,”IJCV, vol. 35, no. 1, pp. 13–32, 1999.

[12] D. M. Gavrila, “A bayesian, exemplar-basedapproach to hierarchical shape matching,”PAMI, vol. 29, no. 8, pp. 1408–1421, 2007.

[13] D. G. Y. Amit and X. Fan, “A coarse-to-fine strategy for multiclass shape detection,”PAMI, vol. 36, no. 12, pp. 1606–1621, 2004.

[14] G. McNeill and S. Vijayakumar, “Hierarchicalprocrustes matching for shape retrieval,” inProc. CVPR, 2006.

[15] L. L. X. Bai and W.-Y. Liu, “Skeleton pruningby contour partitioning with discrete curveevolution,” PAMI, vol. 29, no. 3, pp. 449–462,2007.

[16] E. S. O. Boiman and M. Irani, “In defense ofnearest-neighbor based image classification,”in Proc. CVPR, 2008.

[17] Y. S. A. Frome and J. Malik.

[18] C. G. S. Kirkpatrick Jr. and M. Vecchi, “Op-timization by simulated annealing,” Science,vol. 220, pp. 671–680, 1993.

[19] J. CHen and G. Stockman, “3d free-formobject recognition using indexing by contourfeatures,” PAMI, vol. 24, no. 4, pp. 509–522,2002.

[20] D. Y. X. Bai, X. Yang and L. Latecki,“Skeleton-based shape classification usingpath similarity.” International Journalof Pattern Recognition and ArtificialIntelligence, vol. 22, no. 4, pp. 733–746,2008.