3D Model Multiple Semantic Automatic Annotation For Small Scale Labeled Data Set

Feng Tian1,2, SHEN Xu-kun1 1State Key Laboratory of Virtual Reality Technology

and Systems BeiHang University

Beijing, China Tianfeng80@gmail.com

Liu Xian-mei2,Xie Hong-tao2 2School of Computer and Information Technology

Northeast Petroleum University DaQing, China

liuxianmei@yeah.net

Abstract—Automatically assigning keywords to 3D models is of great interest as it allows one to retrieve, index, organize and understand large collections of 3D models. Most Methods require high sample size for training, so the data quality is in high demand. For small scale labeled data set, we propose a semi-supervised method to realize the 3D models multiple semantic annotation, which needs only a small amount of hand tagged information provided by users. The proposed technique utilizes low-level shape features and the keywords are assigned using a graphed-based label transfer mechanism to expand the training dataset. A weighted metric learning method is used to learn the distance measure from the extended dataset. Then multiple semantic annotation task can be completed on the learned distance measure. The proposed method outperforms the current state-of-the-art methods on the small scale labeled dataset and large unlabelled dataset. We believe that such measure will provide a strong platform to label 3D models when a small amount of labeled models were given.

Keywords-3D model annotation;3D model retrieval;

I. INTRODUCTION With the increasing popularity of 3D applications such as

computer games, a lot of 3D geometry models are being created.To encourage sharing and reuse, techniques that support matching and retrieval of these models are emerging. Several 3D model search engines have been developed. Such as the 3D model search engine at Princeton University [1], the 3D model retrieval system at the National Taiwan University [2], the Ephesus search engine at the National Research Council of Canada [3]. These search engines are all include two search types. One is using traditional text-based retrieval which keywords are extracted from captions, titles, etc. The other type is using content-based retrieval method which search sample is 2D image or 3D object. In contrast, content-based 3D shape retrieval methods, that use shape properties of the 3D models to search for similar models, work better than traditional text-based methods [4]. But compare the 2D image or 3D object, the texture keyword is easier to define and get. The text-based retrieval provides users with a simple and natural interface, so it is friendlier for the user, but the text labels is required [5-6].

3D model auto-annotation is an active subject of research [7]. The goal is to develop methods that can predict for a

new 3D model the relevant keywords from an annotation vocabulary. These keyword predictions can be used either to propose tags for a 3D model, or to propose 3D model for a tag or a combination of tags. Such methods are becoming more and more important given the growing collections of user-provided visual content, 3D scanning equipment, modeling tools and Internet technology. These large-scale collections feed the demand for automatic retrieval and annotation methods. Since the amount of 3D models with more or less structured annotations is also increasing, this allows the deployment of machine learning techniques to leverage this potential by estimating accurate tag prediction models. Most current automatic annotation methods need a large number of models hand tagged with text labels, so the training sample size and quality are in high demand [8]. At the same time manually annotation brought tedious workload, which made the label results imperfect, inaccurate and subjective. Figure 1 show some hand tagged models and labels.

Figure 1. Four hand tagged models.

In this paper, we present a 3D model multiple semantic automatic annotation method based on semi-supervised metric learning(3DSSML), which has achieved a better annotation result when a small amount of labels were given. Section II overviews the process of 3DSSML briefly. Section III introduces our graph-based semantic label propagation method to expand the training dataset. Section IV illustrates our weighted metric learning method to learn the distance measure. Section V introduces our multiple semantic annotation method for the unlabeled models. Finally we give the conclusion.

II. THE OVERVIEW OF 3DSSML In traditional 3D model annotation or classification

approaches, one uses only a labeled set to train the classifier. Labeled 3D models however are often difficult, expensive, or time consuming to obtain, as they require the efforts of

2011 International Conference on Virtual Reality and Visualization

978-0-7695-4602-5/11 $26.00 © 2011 IEEE

DOI 10.1109/ICVRV.2011.54

193

experienced human annotators. Meanwhile unlabeled data may be relatively easy to collect, but there has been few ways to use them. Semi-supervised learning addresses this problem by using large amount of unlabeled data, together with the labeled data, to get better annotation result. Because semi-supervised learning requires less human effort and gives higher accuracy, it is of great interest both in theory and in practice.

The process flow by 3DSSML is shown in Figure 2. The corpus is comprised of a small amount of hand tagged models users provided and a large number of unlabeled models. Firstly, the feature of 3D models was extracted, and the process of dimension deduction is needed. Secondly, we make full use of the unlabeled models to expand the training dataset (known as label propagation) and the label confidence was computed. Thirdly, a new distance metric considered label confidence as well as the correlation between features is learned. Lastly, for each model in unlabeled models collection, we label it by multiple semantic annotation strategy.

Figure 2. 3D model semantic annotation process of 3DSSML.

III. THE SEMI-SUPERVISED LABEL TRANSFER Since the amount of labeled models is not sufficient for

automatic annotation, we need take full use of labeled and unlabeled models to expand the amount of labeled models. In this section, we present graph-based semi-supervised learning method. The graph-based semi-supervised learning has become the mainstream of semi-supervised learning because of its efficiency[9-11]. Let )},(),{( ||||11 LL yxyxL ⋅⋅⋅= be the labeled data, where ix denotes the model and

iy denotes i-th model’s semantic label collection, each sample ix is a point in a d-dimensional feature space, T⊂iy , }{ ||1 TT λλ ⋅⋅⋅= denotes the collection consisting of all labels. },,{ 1|| nL xxU ⋅⋅⋅= + denotes the unlabeled model. The model ix is represented by the point

ix in the d dimension feature space. A graph is defined with each vertex corresponding to each sample in UL ∪ , and the weighted edges reflect the similarity between neighboring samples. We assume that graph },{ EVG = is represented by an nn × symmetric similarity matrix W .Each element of the matrix can be formally defined by RBF kernel function as equation (1):

���

����

� −−= 2

2||||exp

αji

ij

xxw (1)

where ijw denotes the similarity between model ix and jx

and )1,0(∈ijw The distance used affects the ranking performance, we compared several distance measures for their performance, such as cosine measure, K-L divergence,

normLk − having k=0.5,1 and 2, and we use the normL −5.0 for the experiment followed; α represents a particular constant. The objective of graph-based SSL methods is to associate a real-valued label

ifwith vertex iv to capture the

probability that ix is relevant to a certain concept. Regularization framework is used to ensure that

[ ]Tnffff ,...,, 21= is consistent with L and that nearby

samples should have similar labels. Define the loss function as equation (2):

� �= =

−+−=n

ji

n

iiiji

i

ij yfffdw

fQ1, 1

22 ))()((21)( μ (2)

where � ==

n

j iji wd1

.

The first term of )( fQ describes the total variation of labels with respect to the local structures, so called smoothness term. This term indicates that a good predicting function should not change too much between neighboring points. The second term is regularization term. The trade-off between these two competing terms is captured by a positive parameter μ . Note that the regularization term contains both the labeled and unlabeled data.

Rewrite )( fQ into the following matrix form as equation(3):

)}()()({21)( 1 yfyffWDIffQ TT −−+−= − μ (3)

where D is a diagonal matrix with id as its (i,i)-element and I is the identity matrix. Minimizing )( fQ could obtain the optimal prediction function )(minarg* fQf

f= .

Differentiate )( fQ with respect to f , we have :

)()(|)( **1* yffWDI

ffQ

f−+−=

∂∂ − μ (4)

bicycle 0.93 pedal cycle 0.85 push-bike 0.78

Feature Extraction

Dimension Reduction

Label Propagation

chair

Small amount of labelled 3D models +A large number of unlabeled models

Extended Collection

chair 0.87 bench 0.73

Weighted metric learning

Multi-semantic annotation strategy

194

which can be transformed to

yfWDu

Iμ

μ+

=+

− −

1)

11( *1 (5)

Set WDS 1−= and μ

α+

=1

1 ,the final result can be

represented as equation(6) : .))(1( 1* ySIf −−−= αα (6)

It has been shown that this form of result can be obtained in an iterative manner [11], which is a label propagation process. For computational efficiency, we use the iterative solution in our approach. Then, the label propagation process is conducted by Algorithm as follows:

1) Construct the similarity matrix W with entries ijw , and form the initial label vector 1|| ×Ty with each entry indicates the initial label of each model;

2) Construct the matrix WDS 1−= in which D is a diagonal matrix with its (p, p)-element indicating the sum of the pth row of W;

3) The semantic label of model ix is expressed by ||1 T× row vector if , if Lxi ∈ , the j-th element is defined as

follows:

�

∉∈

=i

iij yj

yjf

0,1

(7)

That is, the j-th elements of if is 1 if the j-th label in T is one of the models label, and the rest are zero. If Uxi ∈ ,

]1,0[∈ijf .Denote the labels of 3D models

as [ ]Tnffff ,...,, 21= .Iterate ( ) ytSftf )1()(1 αα −+=+ until

convergence, where α is a parameter between 0 and 1; yf =)0( denotes initial label vector.

4) As the similarity matrix eigenvalue meet 10 ≤≤ ke , we can get convergence of the solution as equation (8):

.))(1( 1* ySIf −−−= αα (8) Each element of f will be assigned a real-value which is used to measure the confidence of model label .We can take labels corresponding to the first k largest score as model’s semantic labels.

The algorithm step (3) (4) shows that the unlabeled data’s label is constantly being updated by the label propagation algorithm, and the labeled data is a starting point, the information of label firstly transferred to the nearest neighbors, then to the secondary neighbors. The final state of label propagation is all the vectors of the unlabeled data are no longer changed, that is semantic labels achieves a smooth distribution in all the unlabeled data. Thus each model has the first k semantic labels. We expands the manually labeled data set L to UL ∪ , meanwhile for each label we assigned a confidence value which we interpret as the probability that the label is a relevant to the model. So the relevance of each model in UL ∪ to each label can be described by the triple like this (xi, ‘airplane’, 0.83).

IV. WEIGHTED METRIC LEARNING The above method can extend labeled data set, and we can

learn a new distance metric considered the size of label information as well as the correlation between features with the data in the extended labeled dataset UL ∪ . RCA (Relevant Component Analysis) is a simple and effective distance metric learning method. It can learn a global linear transformation in the same class constraints that users provided. In pattern recognition field its performance is better than the usual Euclidean distance and other distance metrics[12]. An example of RCA applied to toy data is shown in Figure 3.

(a)The fully toy data set

(b)Labeled toy data set

(c) Processed with RCA Figure 3. An illustrative example of RCA applied to toy data

195

But we found that when the amount of labeled information is insufficient, the size, presence of noise and error of the data as shown in Figure 4, the result got from traditional RCA will bias.

(a) small amount of labeled data

(b)wrong labels added to (a) Figure 4. Examples of insufficient labels.

So we propose a method called weighted RCA. The extended labeled dataset and labeling confidence we got are a guarantee of the algorithm’s validity.

We firstly normalize the label confidence of each label in T , then get a || T -dimensional diagonal matrix of confidence:

],...,,,[ ||321 TwwwwDiagonalW = (9) where iw is mean confidence of all the models described

by i-th label in T . So we can use weighted covariance matrix instead of

centralized covariance matrix of RCA. We summarize the process of 3DSSML as follows: 1) Given the data set )},)...(,{( ||||11 TT yxyxL = which is

comprised of || L models, we propagate the label of the model to the unlabeled model setU , so we get the label’s confidence of each model which has been propagated (see section 3).

2) Each label’s confidence are normalized, and a |||| TT × diagonal matrix of confidence is generated, the weighted covariance matrix of all the labeled model are calculated:

Tmeancic

T

c

n

imeancic xxWxx

nC

i

)()(1,

||

1 1, −

= =− −−= �� (10)

where icx , denotes i-th model in feature space described by

c-th label. meancx − is mean point in feature space described by c-th label.

3) Calculate 1−C as a mahalanobis distance metric: )()(),( 21

12121 xxCxxxxd T

RCAweighted −−= −− (11)

V. MULTIPLE SEMANTIC ANNOTATION FOR THE UNLABELED MODELS

Given an unlabeled 3D model newX we wish to assign labels from the set of all possible labels }...{ ||1 TT λλ= to newX . Specifically, for each label we wish to assign a confidence value which we interpret as the probability that

iλ is a relevant label for newX .So we start with a geometric shape similarity metric and find the neighbors of

newX within some distance threshold. Note that the distance threshold is allowed to be a function of the model, which allows for adaptively defining the threshold based on the density of models in a given portion of the descriptor space. We take

2)),(1(

)P(

ineighbournewRCAweighted

ineighbournew

XXd

XX

−−

−

−

=≈ (12)

to be an estimate of the probability that newX and

ineighbourX − represent the same type of model and therefore should have similar text labels. Then given our unlabelled model newX , a possible text label iλ , and a neighbor

ineighbourX − from the extended labeled data set UL ∪ (see section 3), the probability that newX should have the label is

),()(),(

ineighbouri

ineighbournew

newi

XCXXPXC

−− ∧≈=

λλ

(13)

Where ),( newi XC λ denotes the confidence of

label iλ Intuitively this means that the probability that iλ is appropriate for newX is the probability that it is appropriate for ineighbourX − and that newX and ineighbourX − are similar enough to share labels. ),( new

i XC λ can be thought of as measuring how much we trust the original annotation on ineighbourX − . When considered over the full set of k neighbors this generalizes to

),()(

),(

1jneighbour

ijneighbournew

k

j

newi

XCXXP

XC

−−=

∧≈

=

λ

λ

� (14)

By analogy to the TF-IDF method from text processing we reweight these probabilities such that:

�� ∪∈

−

∪⋅

=

ULX ki

j newj

newi

newi

idftf

kXC

ULXC

XC

XC

),(||log

),(),(

),(

λλλλ

(15)

For each unlabeled model, we get a vector of probabilities for each semantic label. We choose the TOP-N labels to describe the model.

196

VI. EXPERIMENT AND RESULT As the research on automatic annotation of 3D models just

started, there is no established benchmark for an objective evaluation. Ideally, one should use a large enough corpus, In addition to the database, one also needs a method to evaluate the tagging performance numerically and objectively. To evaluate the proposed method, our experiments were performed on a database containing 1253 3D models, which were collected from the Princeton Shape Benchmark (PSB) [13]. In order to evaluate the methods described in this paper, 785 were semantically hand tagged with text labels. Figure 5 shows some automatic annotation samples.

Figure 5. automatic labeling samples.

In the experiment, this paper mainly uses the depth buffer method to extract the 3D models’ feature (438-dimensional feature vector) in the model base[14]. The shape feature we adopted is not the most accurate, Because we stresses the learning and annotation process in this paper, But the more accurate feature such as the work in [15] or other feature extraction method can also substitute the depth buffer method we used. Then We performed a PCA over the descriptors, and kept only the top 20 dimensions. This preserved 90% of the original variance and led to much faster neighbor search. In this paper, we use " Average Precision" VS “Percentage of each tag labeled” to evaluate both automatic labeling process and retrieval process, figure 7 lists the average retrieval precision of five times. These types of labeling methods, including: Euclidean distance metric method, a typical supervised classification learning method (SVM algorithm and the Euclidean distance), RCA distance metric method (RCA algorithm and mahalanobis distance) and 3DSSML. Figure 6 shows that the proposed method has the higher labelling precision when there is a small amount of label information. Among them, the kernel function of the supervised labelling method SVM adopted RBF kernel[16]; the distance metric function used Euclidean distance. Since SVM requires a large number of training data, so if we select a few data sets for training, the labelling result

was not be accurate and led to low retrieval precision. We also test the validity of the proposed method on a small labelled information.

Figure 6. Comparison of average retrieval precision

Table I respectively shows the average retrieval efficiency of various methods in the case of very few labelled data(label 1, 2, 3 and 4 models for each label).Only the first 16 retrieval results will be taken into account.

TABLE I. COMPARISON OF THE RETRIEVAL EFFECTIVENESS WITH A SMALL AMOUNT OF LABELS

methods Labeled models

per label

Precision (%)

Recall (%)

Supervised method

SVM and Euclidean distance

1 19.51 9.51 2 36.65 14.28 3 51.73 18.41 4 66.87 24.74

Semi-supervised

method

RCA

1 40.12 15.37 2 46.22 16.83 3 48.37 17.46 4 68.79 25.82

3DSSML

1 76.26 28.51 2 81.58 28.44 3 82.41 37.13 4 85.82 42.39

We compared “RCA automatic annotation method” with “3DSSML”. Table II lists the comparison of the first 9 retrieval results.

TABLE II. COMPARISON OF THE FIRST 9 RETRIEVAL RESULTS USING RCA METRIC LEARNING METHOD AND 3DSSML

Search target RCA automatic annotation 3DSSML automatic

annotation

chair

197

In this experiment, we labeled one model per semantic label and the average retrieval efficiency are Precision=76. 3% , Recall= 28.5%.The result shows this proposed semi-supervised distance metric learning methods has a better retrieval results in the case of very few labeled data information.

VII. CONCLUSION AND FUTURE WORK In this paper, we have proposed a multiple semantic

automatic annotation method (3DSSML) for small scale 3D models labeled data set. The method acquires a small amount of hand tagged information provided by users, and the semi-supervised semantic label propagation takes full use of unlabeled models to expand the training dataset. The expanded collection increase in the number of labeled models; meanwhile labeling confidence we got can describe the semantic relevance of the label, on the basis of the above two points, Weighted-RCA method can effectively resolve the traditional RCA learning bias caused by the insufficient amount of labeled data or inaccurate labeling information during label transfer. The result on the Princeton Shape Benchmark shows that 3DSSML get a better retrieval results and performance, so the method not only reduce hand tagged information, but also improves the retrieval accuracy in the case of very few labeled data information.

In the future, we definitely would like to employ more “realistic” corpus, e.g., larger size, presence of noise and error, etc., to evaluate the proposed algorithm. We also would like to enhance the method by using a combination of multiple shape features.

ACKNOWLEDGMENT This work is supported and funded by National High

Technology Research and Development Program 863 of China (No. 2009AA012103), the Science and Technology Research Program from Education Department of Heilongjiang Province of China(No. 12511011). We also would like to thank the anonymous reviewers for their helpful comments and suggestions.

REFERENCES [1] Princeton Shape Retrieval and Analysis Group, 3D Model

SearchEngine,http://shape.pe.cs.princeton.edu/search.html, 2008.

[2] Chen Ding-yun, Tian Xiao-per, Shen Yu-te, “On visual similarity based 3D model retrieval,” Eurographics, 2003, pp. 22-25.

[3] Paquet E, Murching A, Naveen T, Tabatabai A, Rioux M, “Description of shape information for 2-D and 3-D objects,” Signal Process Image Commun, 2000,pp. 103–122.

[4] JWH Tangelder,RC Veltkamp, “A survey of content based 3D shape retrieval methods,”Shape Modeling Applications,2004, pp.145-156.

[5] Meng Z, Atta B, “Semantic-associative visual content labeling and retrieval: A multimodal approach”,Signal Processing: Image Communication, 2007, pp.569-582.

[6] G.Leifman, “Semantic-oriented 3d shape retrieval using relevance feedback,” The Visual Computer (Pacific Graphics)

2005,21(8-10),pp.865—875. [7] Ryutarou Ohbuchi Akihiro Yamamoto Jun Kobayashi

“Learning semantic categories for 3D Model Retrieval,”Proc.ACM MIR,2007,pp.31-40.

[8] Benjamin Bustos,“An Experimental Effectiveness Comparison of Methods for 3D Similarity Search,” International Journal on Digital Libraries.2006,6(1),pp.39-54.

[9] O. Chapelle, A. Zien,B. Scholkopf, “Semi-supervised Learning,” MIT Press,2006.

[10] D.Zhou, O.Bousquet, T.Lal, J. Weston, B.Scholkopf, “Learning with local and global consistency,” Advances in Neural Information Processing Systems, Vol. 16, 2003,pp.321-328.

[11] X.Zhu, Z.Ghahramani,J.Lafferty, “Semi-supervised learning using Gaussian fields and harmonic functions,” Proceedings of International Conference on Machine Learnings,2003,pp.912-919.

[12] Noam S, Tomer H, Daphna W, “Adjustment learning and relevant component analysis,” Proc. European Conference of Computer Vision (ECCV), Copenhagen, 2002,pp.776-790.

[13] S.Philip, M.Patrick, K.Michael, “The Princeton Shape Benchmark,”Shape Modeling International, 2004,pp.388-399.

[14] M.Heczko, D.Keim, D.Saupe, “Methods for similarity search on 3D databases,”Datenbank-Spektrum ( In German) , 2002, 2( 2),pp.54-63.

[15] G. Tam and R. Lau, "Deformable odel retrieval based on topological and geometric signatures," IEEE Trans. on Visualization and Computer Graphics, 13(3), 2007, pp.470-482.

[16] HOU S, LOU K, RAMANIK,“SVM based semantic clustering and retrieval of a 3D model database,” Journal of Computer Aided Design and Application,2005,pp.155-164.

198