011001-1 CHINESE OPTICS LETTERS / Vol. 9, No. 1 / January 10, 2011

Point pattern matching based on kernel partial least squares

Weidong Yan (òòò���ÀÀÀ)1∗, Zheng Tian (XXX ���)1,2, Lulu Pan (���åååååå)1, and Jinhuan Wen (§§§777���)1

1School of Science, Northwestern Polytechnical University, Xi’an 710072, China2State Key Laboratory of Remote Sensing Science, Institute of Remote Sensing Applications, Chinese Academy of Sciences,

Beijing 100101, China∗Corresponding author: weidongyan@qq.com

Received July 5, 2010; accepted August 3, 2010; posted online January 1, 2011

Point pattern matching is an essential step in many image processing applications. This letter investigatesthe spectral approaches of point pattern matching, and presents a spectral feature matching algorithmbased on kernel partial least squares (KPLS). Given the feature points of two images, we define positionsimilarity matrices for the reference and sensed images, and extract the pattern vectors from the matricesusing KPLS, which indicate the geometric distribution and the inner relationships of the feature points.Feature points matching are done using the bipartite graph matching method. Experiments conducted onboth synthetic and real-world data demonstrate the robustness and invariance of the algorithm.

OCIS codes: 100.0100, 100.2000.doi: 10.3788/COL201109.011001.

Point matching and correspondence problems arise invarious application areas such as computer vision, pat-tern recognition, machine learning, and so on. In 1993,Cox et al. gave an overview of the different types oftransformations and methods[1]. The advantage of pointset representations of shapes over other forms such ascurves and surfaces is that the point set representationis a universal representation of shapes regardless of thetopologies of the shapes[2].

Given two or more images represented using points,we can match these point patterns to determine a trans-formation between coordinates of the point sets. Suchtransformations capture changes in the geometry charac-terized by the given points. Feature points are the sim-plest form of features, which are basically represented bythe point locations. Feature points often serve as the ba-sis upon which other more sophisticated representations(such as curves and surfaces) are built. They can also beregarded as the most fundamental of all features. Topo-logical and geometrical relations between features containimportant information to constrain the large space of pos-sible mappings between features[3]. Feature points can bematched either by considering the radiometric propertiesof surrounding pixels or the geometric distribution of thewhole set of feature points across the image[4]. Spectralgraph theory is a term applied to a family of techniquesthat aim to characterize the global structural propertiesof graphs using eigenvalues and eigenvectors of similar-ity matrices[5,6]. The spectra can indicate the geometricdistribution of feature points. Graph spectral methodshave been extensively used for correspondences betweenfeature point sets[7−9]. Scott et al. first used a Gaus-sian weighting function to build an inter-image similaritymatrix between feature points in different images beingmatched, and performed singular value decomposition onthe similarity matrix to get correspondences from thesimilarity matrix’s singular values and vectors[7]. How-ever, this method fails when the rotation or scaling be-tween the images is too large. To overcome this problem,Shapiro et al. constructed intra-image similarity matri-ces for the individual point sets being matched with theaim of capturing the relational image structure[8]. The

eigenvectors of the individual similarity matrices are usedin the matching process. This method projects the indi-vidual point sets into an eigenspace and seeks matchesby looking for the closest point correspondence. Wanget al. investigated the performance of the kernel princi-pal component analysis (PCA) using a polynomial kernelfunction for solving the point correspondence problem[9].Such approaches characterize the graphs by their domi-nant eigenvectors. However, these eigenvectors are com-puted independently for each graph, thus they often donot capture the co-salient structures of the graph. Thekernel partial least squares (KPLS) approach helps ex-tract representations from the two images containing rel-evant information needed for the matching of the partic-ular pair of images.

This letter presents a method of spectral feature match-ing based on KPLS. Using KPLS components, the spec-tral features are constructed with information on theposition similarity matrices, invariant to translation,scale, and rotation, and very suitable for feature-basedmatching.

In general, intrinsic point representations can be ob-tained via the position similarity matrix, specified by asymmetric affinity matrix S = {sij}, where sij ≥ 0 char-acterizes the similarity between points xi and xj . Onemay view the position similarity matrix S as a data vec-tor whose n rows represent n-dimensional data points.

Although the position similarity matrices contain con-siderable shape information, one cannot compare suchrepresentations for two point sets directly without properpoint mapping. The high dimensional representationsmay contain a great deal of redundancy, resulting in un-necessarily high computational costs[10]. These obser-vations naturally lead us to consider transforming twopoint sets into some information-preserving subspaces.This can be accomplished through KPLS on the positionsimilarity matrices.

Partial least squares (PLS) method, which was ini-tially developed by Wold et al.[11], has been a tremen-dously successful method for data analysis in the chemo-metrics and chemical industries. The robustness ofthe generated model also makes the PLS approach a

1671-7694/2011/011001(5) c© 2011 Chinese Optics Letters

January 10, 2011 / Vol. 9, No. 1 / CHINESE OPTICS LETTERS 011001-2

powerful tool for dimensionality reduction, discrimina-tion, and classification technique; it is being applied tomany other areas such as process monitoring and imageprocessing[12−14]. In its general form, PLS creates com-ponents using the existing correlations between differentsets of variance while keeping most of the variance of bothsets. KPLS is a modification of PLS through the use ofkernels. Following the theoretical and practical resultsreported in Ref. [14], KPLS seems to be preferred to thekernel PCA when a feature space dimensionality reduc-tion with respect to data discrimination is employed.

We consider a general setting of the PLS algorithmto demonstrate the relation between two data sets. AnN -dimensional vector of variables is denoted as x =(x1, · · · , xN ) in the first block of data, and similarly,y = (y1, · · · , yN ) denotes a vector of variables fromthe second set. Observing the n data samples fromeach block of variables, PLS decomposes X = (xij)n×N ,Y = (yij)n×N into the form

X = TPT + F,

Y = UQT + G,

where T and U are (n × r) matrices of the extracted rcomponents, and the (N × r) matrices P and Q repre-sent the matrices of the projections. The n×N matricesF and G are the matrices of the residuals. The PLSmethod, which in its classical form is based on the non-linear iterative partial least squares (NIPALS) algorithm,finds the projection axes w and c such that[11]

max S = tTu = (Xw)T(Yc) = wTXTYc

s.t.

{wTw = ‖w‖2 = 1 .

cTc = ‖c‖2 = 1(1)

The solution to this optimization problem is given by thefollowing eigenvalue problem[15]:

XTYYTXw = λw,

where λ is the eigenvalue associated with w. The com-ponent of X is then given as t = Xw.

Similarly, the extraction of the component of Y is givenas

XXTYYTt = λt, (2)

u = YYTt. (3)

Like other kernel-based algorithms, KPLS is based onmapping the original data into feature space, in which alinear regression function is constructed. We consider anonlinear transformation of x into a feature space F. Inthis case, the projection axes w and c cannot be com-puted. Using the straightforward connection between areproducing kernel hilbert space and F, Rosipal et al.extended the linear PLS model into its nonlinear kernelform[16]. This extension effectively represents the con-struction of a linear PLS model in F.

The feature space F could have an arbitrarily large,possibly infinite dimensionality related to the data spaceby a possibly nonlinear map φ : x → φ(x). Φ denotesthe matrix of the mapped data space φ(x) into a fea-ture space F. Using the kernel method, we can obtain

K = ΦΦT, where Kij = K(xi, xj) is the Gram matrix.Similarly, we consider a mapping of the second set of

variables y into a feature space F1, and denote it as Ψ,the matrix of the mapped data space ψ(y) into a fea-ture space F1. Analogous to K, we define the kernelGram matrix as K1 = ΨΨT, given by the kernel func-tion K1(·, ·). We can use K = 〈Φ,Φ〉 and K1 = 〈Ψ,Ψ〉instead of XXT and YYT because XXT = 〈X,X〉 andYYT = 〈Y,Y〉 are inner product forms. Using this no-tation, the estimates of t and u can be reformulated intoits nonlinear kernel variant (KPLS) from Eqs. (2) and(3):

KK1t = λt,u = K1t.

When the data in the feature space do not contain zeros,the mean 1

n

∑i

Φ(xi) is subtracted from all points. This

leads to a slightly different expression

K =(In − 1

n1n1T

n

)K(In − 1

n1n1T

n ),

where 1n = (1, · · · , 1)T.Similarly, K1 = (In − 1

n1n1Tn )K1(In − 1

n1n1Tn ).

We select feature point sets from the reference im-age and the sensed image, and denote them as X =(x1, x2, · · · , xn) and Y = (y1, y2, · · · , yn), respectively.Our objective is to establish a one-to-one point corre-spondence between the two data sets.

Using the feature point sets, we construct position sim-ilarity matrices by the Gaussian kernel function, (Sx)n×n

and (Sy)n×n:

(Sx)ij = exp[− d(xi, xj)2

σ2x

],

and (Sy)ij = exp[−

d(yi, yj

)2

σ2y

],

where d(xi, xj) is the Euclidean distance between thefeature points xi and xj , and σx, σy are the adjustableparameters. Every feature point corresponds to anndimensional feature vector, xi → (Sx)i, yj → (Sy)j .

Theorem 1. Under criterion (1), the number of com-ponents is rank(ST

x SySTy Sx) (the number of non-zero

eigenvalues of matrix STx SyST

y Sx) pairs at most. r(≤rank(ST

x SySTy Sx)) pairs of components are composed of

vectors selected from the eigenvectors corresponding tothe first r maximum eigenvalues of the eigenequations

SxSTx SyST

y T = λ2T, (4)

SySTy SxST

x U = λ2U, (5)

where T = [t1, · · · , tr], U = [u1, · · · , ur].See Ref. [17] for detailed proof.Similarly, let K = SxST

x , K1 = SySTy . Thus, we can

derive Theorem 2 from Theorem 1 easily.Theorem 2. Under criterion (1), the number of com-

ponents is rank(STx SyST

y Sx) (the number of non-zeroeigenvalues of matrix ST

x SySTy Sx) pairs at most. r(≤

rank(STx SyST

y Sx)) pairs of components are composed of

011001-3 CHINESE OPTICS LETTERS / Vol. 9, No. 1 / January 10, 2011

vectors selected from the eigenvectors corresponding tothe first rmaximum eigenvalues of eigenequations

KK1T = λT,

K1KU = λU,

where T = [t1, · · · , tr], U = [u1, · · · , ur].The feature point xi in the reference image corresponds

to the KPLS pattern vector (T)i = [ti1, ti2, · · · , tir],which are the projections in the r-dimensional eigenspacespanned by T. The feature point yj in the sensed im-ages corresponds to the KPLS pattern vector (U)j =[uj1, uj2, · · · , ujr], which is the projections in the r-dimensional eigenspace spanned by U. Thus, if thearbitrary numbering of two feature points in an imageis changed, their KPLS pattern vectors simply changepositions in T(or U), and the matching of the featurepoints can be converted to the matching of the KPLSpattern vectors. The Euclidean distance is invariant tothe similarity transformation[10]. Hence, the KPLS pat-tern vectors have invariance.

In the last decade, numerous techniques have beenproposed to tackle the feature point matching prob-lem, such as iterative closest point (ICP)[18], graphmatching[19−21], and cluster approach[22]. Among thesemethods, the ideas of the graph matching algorithm arevery attractive. In graph matching, feature points aremodeled as graphs, and feature matching amounts tofinding a correspondence between the nodes of differentgraphs. In this letter, bipartite matching method[23] isapplied in feature matching. We extract feature pointsfrom the reference and sensed images. The KPLS patternvectors (T and U) are then computed for the featurepoint sets as vertices for a bipartite graph. We applybipartite graph matching algorithms to this graph toobtain the set of edges in the optimal matching, whichrepresents the correspondence.

To make the algorithm more robust, the mismatchingfeature points can be eliminated by the local continu-ity constraint (the continuity constraint requires thatneighboring points in the reference match neighboringpoints in the sensed). This constraint is reasonable formost images. We can examine the correspondences ofthe neighbors of the matched interest points to eliminatemismatching.

Now we investigate the performance of the method ofpoint pattern matching based on KPLS. In experiments,the robustness of the KPLS approach described abovewith random point sets is shown, the invariance of theKPLS pattern vectors on synthetic images is verified, thematching performances of the KPLS approach describedabove and the method of Ref. [9], are compared, and theremote sensing image. In addition, we use this algorithmto solve the non-rigid mapping.

We investigated the effect of the controlled affine skewof the point sets. The reference point set was randomlygenerated. Then, the reference point set was transformedby parameters (translation x=0.1, translation y=0.1,rotation angle=30◦, scaling factor=1.2) to obtain thesensed data set. Figure 1 shows the matching results.We then focused on the performance of the algorithmswhen the data were under affine transformations andcontained uncertainties such as outliers and noise. For

this purpose, we added noise to the sensed data. Thereference data set is the same as that used above, andfive outliers were added to the sensed data set. Figure2 shows the matching results. It can be seen that theKPLS algorithm presents a strong ability to eliminateincorrect matches.

To provide more quantitative evaluations, we alsotested the algorithm on synthetic images. Here, we havematched images from a gesture of a hand. The feature

Fig. 1. (a) Reference data set and (b) sensed data set withoutoutliers, and (c) the match between the two data sets foundwith our implementation. The circles in (c) are the referencedata and the crosses are the sensed data.

Fig. 2. (a) Reference data set and (b) sensed data set withoutliers, and (c) the match between the two data sets foundwith our implementation.

January 10, 2011 / Vol. 9, No. 1 / CHINESE OPTICS LETTERS 011001-4

points in these images are points of maximum curvatureon the outline of the hand. Figure 3 shows the finalconfiguration of correspondence matches obtained usingour method. From these experiments, we can see thatthe KPLS pattern vectors are invariant to rotation andscaling.

We tested our algorithm on the Carnegie Mellon Uni-versity (CMU) house sequence, and compared the per-formance of our method with that of the method of Ref.[9]. The images used in our study correspond to differentcamera viewing directions. The corner points in each

Fig. 3. (Point matching result on the hand images. Top row:the correspondences between the hand-rotation (left) and theregistration result (right). Bottom row: the correspondencesbetween the hand-scaling (left) and the registration result(right).

Fig. 4. Point matching result on the CMU house sequence.Top row: our approach. Bottom row: the kernel PCAmethod.

Fig. 5. (a) Band-1 and (b) band-2 images.

image were detected by the Harris detector. The firstframe was tested against the remaining frames. Figure4 shows the correspondences when we matched the firstframe to the fourth and the sixth frames. The experi-mental results are summarized in Table 1. From theseresults, our method clearly performs better than Wangand Hancock’s method.

To test our algorithm on a real world situation, we ap-plied our method to a two-dimensional (2D) image reg-istration with an affine model. We took two images from

Table 1. Summary of the Experimental Results onthe CMU House Sequence

Image 1−2 1−3 1−4 1−5 1−6

Correctly Matched28 22 28 28 26

(Our Approach)

Rate of Matched100 79 100 100 93

(Our Approach) (%)

Correctly Matched24 20 22 22 20

(Method of Ref. [9])

Rate of Matched86 71 79 79 71

(Method of Ref. [9]) (%)

Fig. 6. (a) Correspondences of feature points between the im-ages and (b) registration result computed with the proposedapproach.

Fig. 7. (a) Reference image and (b) sensed image.

011001-5 CHINESE OPTICS LETTERS / Vol. 9, No. 1 / January 10, 2011

Fig. 8. Correspondences of points between the images.

the same area. The feature points in each image were de-tected by a Harris detector. We then tried to match themusing our implementation. Figure 5 shows the two multi-spectral images (band-1 and band-2). Figure 6 shows ourregistration result.

We tested our algorithm on non-rigid transformations.The reference image that we chose comes from a Chinesecharacter (blessing), which is a rather complex pattern.Figure 7 shows the character images and Fig. 8 showsthe results of the matching.

In conclusion, we investigate the spectral approaches tothe problem of point pattern matching. First, we showhow KPLS can be effectively used for solving the pointpattern matching problem. We use the KPLS patternvectors of feature point sets from both reference andsensed images to establish their correspondences. TheKPLS pattern vectors indicate the geometric distribu-tion of the feature points, and are invariant to transla-tion, scale, and rotation. We then use robust methods forthe points correspondence by the continuity constraint.Results show that the method can deal with non-rigidtransformation. Further work has to be done in order toobtain more robust results using more matching featurepoints to get better mapping approximations and to workwith feature point sets of different sizes.

This work was supported by the Northwestern Poly-technical University Doctoral Dissertation InnovationFoundation (No. CX200819), the National NaturalScience Foundation of China (Nos. 10926197 and60972150), and the Science and Technology InnovationFoundation of Northwestern Polytechnical University(No. 2007KJ01033).

References

1. G. S. Cox and G. De Jager, in Proceedings of SouthAfrican Symposium on Communications and Signal Pro-cessing 243 (1993).

2. H. Guo, A. Rangarajan, and S. Joshi, “Diffeomorphicpoint matching” in Handbook of Mathematical Mod-els in Computer Vision (Springer, Heidelberg, 2005)pp.205−220.

3. J. You and P. Bhattacharya, IEEE Trans. Image Process.9, 1547 (2000).

4. P. Dare and I. Dowman, ISPRS J. Photogramm. RemoteSens. 56, 13 (2001).

5. F. R. K. Chung, Spectral Graph Theory (American Math-ematical Society, Providance, 1997).

6. M. Carcassoni and E. R. Hancock, Pattern Recognition36, 193 (2003).

7. G. L. Scott and H. C. Longuet-Higgins, Proc. R. Soc.Lond. B 244, 21 (1991).

8. L. S. Shapiro and J. M. Brady, Image and Vision Com-puting 10, 283 (1992).

9. H. Wang and E. R. Hancock, LNCS 3138, 361 (2004).

10. V. Jain and H. Zhang, in Proceedings of 2006 IEEE Inter-national Conference on Shape Modeling and Applications118 (2006).

11. S. Wold, A. Ruhe, H. Wold, and W. J. Dunn III, SIAMJ. Sci. Stat. Comput. 5, 735 (1984).

12. D. V. Nguyen and D. M. Rocke, Bioinformatics 18, 39(2002).

13. Q.-S. Sun, Z. Jin, P.-A. Heng, and D.-S. Xia, LNCS3686, 268 (2005).

14. M. Barker and W. Rayens, J. Chemometrics 17, 166(2003).

15. A. Hoskuldsson, J. Chemometrics 2, 211 (1988).

16. R. Rosipal and L. J. Trejo, Machine Learning Res. 2, 97(2002).

17. W. Yan, Z. Tian, L. Pan, and M. Ding, Chin. Opt. Lett.7, 201 (2009).

18. P. J. Besl and N. D. McKay, IEEE Trans. Pattern Anal.Mach. Intell. 14, 239 (1992).

19. T. Jebara and V. Shchogolev, LNAI 4212, 679 (2006).

20. T. S. Caetano, J. J. McAuley, L. Cheng, Q. V. Le, andA. J. Smola, IEEE Trans. Pattern Anal. Mach. Intell.31, 1048 (2009).

21. C. Leng, Z. Tian, J. Li, and M. Ding, Chin. Opt. Lett.7, 996 (2009).

22. S.-H. Chang, F.-H. Cheng, W.-H. Hsu, and G.-Z. Wu,Pattern Recognition 30, 311 (1997).

23. Y. Wang, F. Makedon, and J. Ford, in Proceedings of the26th Annual International Conference of the IEEE En-gineering in Medicine and Biology Society 2, 2972 (2004).