IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 2, FEBRUARY 2014 469

A Robust Point-Matching Algorithm forRemote Sensing Image Registration

Kai Zhang, XuZhi Li, and JiuXing Zhang

Abstract—Feature point matching is a critical step in feature-based image registration. In this letter, a highly robust feature-point-matching algorithm is proposed, which is based on thefeature point descriptor calculated by the triangle-area repre-sentation (TAR) of the K nearest neighbors (KNN-TAR). Theaffine invariant descriptor KNN-TAR is used to find the candidateoutliers, and then, the real outliers will be removed by the localstructure and global information. The experimental results showthat the proposed method can remove the outliers from the initialmatching result even when the outliers are of high proportion.Compared with graph transformation matching and restrictedspatial-order constraints, KNN-TAR outperforms these methodswith higher stability and precision.

Index Terms—Affine invariant descriptor, image registration,K nearest neighbor (KNN), triangle-area representation (TAR).

I. INTRODUCTION

IMAGE registration is a key step in many areas of remotesensing image processing, i.e., image fusion, image mosaic,

change detection, and so on. The aim of image registration is tofind the optimization transformation between the reference andsensed images, which are taken at different times, from differentviewpoints, and/or by different sensors. Most of the feature-based registration methods consist of the following steps: fea-ture detection, feature matching, transform model estimation,image resampling, and transformation.

Feature point matching means to establish correspondencebetween the control points extracted from two images. Thechallenges of feature point matching are as follows: the outliersappearing in only one image, some control points slightlydisplaced from their true positions because of noise, large affinetransformation, or other factors [1]. Currently, there are manypoint-matching algorithms, most of which are based on localfeature similarity, spatial relations, or the combination of theprevious two aspects [8]. The research in this letter aims atfeature point matching by spatial relation, which has beenwidely studied but remains a challenging task.

Spatial relations including geometric constraints and neigh-boring structures are used to distinguish correct correspon-

Manuscript received December 17, 2012; revised March 19, 2013 andMay 4, 2013; accepted June 6, 2013. Date of publication July 9, 2013; dateof current version November 25, 2013. This work was supported by China’smanned space project under Grant Y1141401SN.

K. Zhang is with the Academy of Opto-Electronics, Chinese Academy ofSciences, Beijing 100094, China, and also with the University of ChineseAcademy of Sciences, Beijing 100049, China (e-mail: zhangkai@csu.ac.cn).

X. Li and J. Zhang are with the Technology and Engineering Center forSpace Utilization, Chinese Academy of Sciences, Beijing 100094, China.

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/LGRS.2013.2267771

dence from incorrect ones. Random Sample and Consensus(RANSAC) [2] is a classic method to estimate the parametersof the transformation and to establish correspondence betweenthe points using the model. This algorithm performs very wellto remove the outliers when there are no more than 50%outliers. Based on the affine invariant property of triangle-arearepresentation (TAR) [3], Li proposed a new algorithm calledrobust sample consensus judging, which can be embedded intoRANSAC and improve the computational efficiency [4].

Iterative Closest Point [5] is another method commonly used,but this method needs a good initial estimation to guarantee acorrect solution.

Point structure has also been used in feature point matching.The neighborhood structures of points are usually well pre-served due to the physical constraints. Zheng and Doermannformulated point matching as an optimization problem, whilelocal neighborhood structures are preserved during matching[6]. However, this method does not perform well due to theeffect of large rotation or noise. Aguilar proposed a methodcalled Graph Transformation Matching (GTM). In this method,a K nearest neighbor (KNN) graph with the restriction ofaverage distance is constructed for each point, and the ver-tices which introduce structural dissimilarity between the twoimages are eliminated in each iteration [7]. Liu proposed apoint-matching algorithm based on Restricted Spatial-OrderConstraints (RSOC), which makes use of both local structureand global information in each iteration to remove outliers [8].However, when the KNN of the outliers are all the same, RSOCfailed to remove such outliers.

Although extensive work, as mentioned before, has beendone, feature point matching is still a challenging task, par-ticularly for images with simple patterns, large affine transfor-mation, and low overlapping areas. To resolve the problem ofpoint matching, a novel and robust point-matching algorithm isproposed in this letter, which is based on the TAR value of theKNN (KNN-TAR). Both the local structural and global infor-mation are used to deal with the outliers. The rest of this letteris organized as follows: Section II presents the novel feature-point-matching method based on KNN-TAR. Section III detailsthe experiment and analysis of the algorithm proposed in thisletter. Section IV draws the conclusion.

II. OUTLIER REMOVAL ALGORITHM BASED ON KNN-TAR

In the proposed algorithm, an affine invariant descriptor is de-signed to extract the candidate outliers, which will be confirmedto be real outliers or not by the global information. The KNN-TAR descriptor is defined under the assumption that the nearest

1545-598X © 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

470 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 2, FEBRUARY 2014

neighbors of the outliers will be much more different. Mostof the outliers can be found by KNN-TAR, but a few outliershave the same nearest neighbors. How to remove such outliersis very important to the performance of the proposed algorithm.The algorithm proposed in this letter includes three parts: 1) theKNN-TAR descriptor; 2) process of the candidate outliers; and3) removal of the remaining outliers.

A. KNN-TAR

Based on KNN-TAR, a simple descriptor is proposed todetermine whether a matching pair is a candidate outlier or not.

The TAR value is calculated by the coordinates of threepoints to calculate the size of a triangle [3]

TAR(p1, p2, p3) = abs(x1y2 + x2y3 + x3y1

−x3y2 − x1y3 − x2y1)/2 (1)

where xi and yi are coordinates of pi. The relation between thereference and sensed images is given by[

x′

y′

]=

[a1 a2a3 a4

] [xy

]+

[b1b2

]. (2)

By substituting (2) into (1), we obtain

TAR (p′1, p′2, p

′3) = (a1a4 − a2a3)TAR(p1, p2, p3) (3)

where (pi, p′i; i = 1, 2, 3) are matching pairs. It is clear that the

TAR value can be used to construct the affine invariant variable[4], [9], as defined by

TARR(c1, c2, c3) = TAR(p1, p2, p3)/TAR (p′1, p′2, p

′3) (4)

where ci = (pi, p′i), i = 1, 2, 3, and TAR(p′1, p

′2, p

′3) > β.

When p′1, p′2, p′3 are in one line, TAR(p′1, p′2, p

′3) is equal to

zero. Furthermore, if the TAR value is very small, the accuracyof the TARR will be much lower. Therefore, the parameterβ = 0.2 is set to avoid that. It is obvious that the TARR valueof different matching points should be a constant value if nooutlier exists.

Suppose two feature-point sets extracted from two affinetransformed images, which are denoted by C = (c1, c2, . . . cn),where ci = (pi, p

′i), i = 1, 2, . . . n. C is the initial correspond-

ing point set. In order to find the outliers, the KNN is used tocalculate the TARR value. Define the KNN matching pairs of piand p′i as vertices Vi and Vi′ ; then, Vi = {ci1, ci2, ci3, . . . ciK}and Vi′ = {ci1′ , ci2′ , ci3′ , . . . ciK′}. The value of KNN-TAR iscalculated by

TARRKNN-ratio(i)=TARR(ci, cij , cik)/TARR(ci, cij′ , cik′)(5)

where 1 ≤ i ≤ n, 1 ≤ j ≤ K, 1 ≤ k ≤ K, ij, ik ∈ Vi, and ij ′,ik′ ∈ Vi′ . If TARRKNN-ratio(i) < 1, then

TARRKNN-ratio(i) = 1/TARRKNN-ratio(i). (6)

Obviously, if the test points are all correct matching pairs,TARRKNN-ratio(i)∈ [1, ε]. Otherwise, TARRKNN-ratio(i)>ε.

Fig. 1. KNN-TAR descriptor of a pair of outliers. {pi, p′i} is a pair of outliers.{pik, p′ik|k = 1, 2} and {p′ik, p′′ik|k = 1, 2} are all matching pairs. pi1 andpi2 are the two nearest neighbors of pi, while p′

i1′ and p′i2′ are the two nearest

neighbors of p′i.

Fig. 2. KNN-TAR descriptor of a pairs of outliers which have the same KNN.{pi, p′i} is a pair of outliers. {pik, p′ik|k = 1, 2, 3, 4} are all correct matchingpairs.

As illustrated in Fig. 1, (pi, p′i) is a pair of outliers.

Two nearest neighbors of pi and p′i are selected to calcu-late TARRKNN-ratio(i), where cik = (pik, p

′ik) and cik′ =

(pik′ , p′ik′), k = 1, 2. It is obvious that the value is much largerthan the threshold ε. Therefore, the KNNs of pi and p′i arechosen to calculate the value of KNN-TAR. If all of the KNNsare all correct matching points, the value of TARRKNN-ratio(i)can be used to determine whether ci is an outlier or not.

However, when there are outliers in the KNN of ci, thevalue of TARRKNN-ratio(i) may not be accurate to determinewhether ci is an outlier. Therefore, TARRKNN-ratio(i) is usedto find the candidate outliers. Further measures should be takento make sure whether ci is an outlier. The solution will bediscussed in the following section.

In order to minimize the impact of the outliers in the KNNwhen calculating TARRKNN-ratio(i), the average value is usedinstead

TARRKNN-ratio(i) =1

K − 1

·K−1,K−1∑i1=1,i1′=1

TARR(ci, ci1, ci1+1)/TARR(ci, ci1′ , ci1′+1). (7)

KNN-TAR can be used to detect most of the outliers, but insome extreme cases, the KNNs of the outlier may be all thesame. As illustrated in Fig. 2, the matching pair is an outlierwhile the KNNs are the same with a KNN-TAR value of 1.How to remove such outliers will be discussed in Section II-C.

B. Process of the Candidate Outliers

Motivated by Liu’s outlier filtering strategy, a novel point-matching method is proposed based on the descriptor KNN-TAR. Candidate outliers found by KNN-TAR will be confirmed

ZHANG et al.: ROBUST POINT-MATCHING ALGORITHM FOR REMOTE SENSING IMAGE REGISTRATION 471

Fig. 3. Flowchart of proposed outlier removal algorithm.

to be real outliers or not by the local structure of the singlematching pair and the global information of the whole matchingpairs.

As described before, C is a set of matching pairs extractedfrom the reference and sensed images. The optimal match V̂ isthe solution to the following problem:

V̂ = argminV

E(C, V ) (8)

where V is the match vector used to reduce the computationtime, E(C, V ) =

√∑ni=1 vid(ci, θ) denotes the global trans-

form error, and T (θ) is the global transform model, which isobtained by the remaining points in the point set. d(ci, θ) =‖T (θ, p′i)− pi‖. vi is defined as follows:

vi ={0 ci ∈ outliers1 otherwise.

(9)

In order to distinguish the outliers from C and find an optimalcorrespondence matrix to minimize the transformation error,the KNN-TAR descriptor is used to select the candidate outliersin each iteration, and then, each of the candidates will beconfirmed to be an outlier or not. The whole process is shownin Fig. 3.

In each iteration, the value of TARRKNN-ratio(i) is used toselect the candidate outlier instead of removing the pair directly,since in some extreme cases, most of the KNNs of one correctmatching pair might be outliers. If TARRKNN-ratio(i) > ε,then ci is a candidate outlier. The process of outliers is shownin Fig. 4.

The candidate outliers are arrayed on the order of d(ci, θ),with the candidate which has the largest value of d(ci, θ) beingprocessed first. Either of the following two criteria determineswhether the candidate found by TARRKNN-ratio(i) is an out-lier or not:

1) The global transform error difference between before andafter removing any of the candidate matching pair from

Fig. 4. Pseudocode of the proposed outlier removal strategy.

the remaining point set. Let Ec(i) = abs(Ebf − Eaf (i)),

where Ebf =√

(∑rn

j=r1 d2(cj , θ))/(rn) and Eaf (i) =√

(∑rn

j=r1,j �=i d2(cj , θ′))/(rn− 1). θ is calculated by

the remaining matching points in each iteration, whileθ′ is calculated by the remaining matching points ineach iteration except the candidate outlier ci. rn is thenumber of remaining matching pairs. If Ec(i) > σ, thenci ∈ outliers and the process of the next candidate will goon. Otherwise, the process will be ended.

2) If no outlier is found by criterion 1, d(c1, θ) > Ebf andEbf > σ. Suppose that N candidates with the largestd(ci, θ) value meet the condition that (Ebf − Eaf (N)) <σ, (Ebf − Eaf (N + 1)) ≥ σ, and the d(ci, θ) value ofeach candidate is larger than Ebf , where Eaf (N) is theglobal transform error after removing the N candidateoutliers. Then, the N candidates are all outliers.

C. Removal of the Remaining Outliers

As described at the end of Section II-A, the value of KNN-TAR can be used to find the outliers; however, sometimes,there might exist outliers having the same KNN. Such kind ofoutliers will impact the result of the outlier removal algorithm.After several iterations, the correct matching pairs picked outby KNN-TAR might be removed by the second criterion of theremoval of candidate outliers. Thereby, how to find and remove

472 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 2, FEBRUARY 2014

such kind of outliers plays a crucial role in the performance ofthe outlier removal algorithm.

If the KNN of the outliers are all the same, the value of KNN-TAR will be equal to 1. In order to find the outliers, all thematching pairs with a KNN-TAR value of 1 will be checked bythe value of d(ci, θ). If d(ci, θ) > σs, then ci ∈ outliers. σs isdefined as follows:

σs = α · Ebf . (10)

The outliers that cannot be found by KNN-TAR take only asmall part of the whole outliers. Therefore, the removal of suchoutliers will be performed when the second removal criterionof the candidate outliers is not satisfied. If d(c1, θ) ≤ Ebf , itmeans that outliers which cannot be found by KNN-TAR doexist in the remaining matching pairs. Then, the matching pairswith a KNN-TAR value of 1 will be checked. Together with thecandidate outlier removal criteria, the matching result will bemore precise.

III. EXPERIMENTAL RESULTS AND ANALYSIS

To evaluate the performance of the proposed algorithm,20 typical image pairs are used in the experiment, includingimages with large affine transformation, similar patterns, andlow overlapping areas.

The evaluation criteria are recall, precision, and root meansquare error (RMSE) [10], [11], which are defined as follows:

Recall = true_positives/total_positives

Precision = true_positives/

(true_positives+ false_positives)

RMSE =

√√√√ 1

n

n∑i=1

(‖T (p′i, θ)− pi‖)2

where true_positives is the number of correctly matched pointpairs, false_positives is the number of wrongly matched pointpairs, and total_positives is the total number in the initialpoint sets.

A. Parameter Setting

In KNN-TAR, four of the nearest neighbors are chosenafter removing the neighbors which do not meet the require-ment described in Section II. ε is the upper limit of theTARRKNN-ratio(i) value calculated by correct matching pairs.For correct matching pairs, the value of KNN-TAR will be closeto 1; therefore, ε is set to be 1.1.

In the outlier removal algorithm, σ is used to determinewhether the initial matching points are outliers or not. Theproper value of σ is reached by the experimental result of threetypical image pairs. As shown in Fig. 5(d), when σ > 1, themean precision value is less than 1. With the increase of σ,the recall value decreases. As a tradeoff between the recall andprecision values, σ is set to be 1. σs is used to find the outliersthat cannot be found by KNN-TAR. Such kind of outliers onlytakes a small part of the whole outliers or they do not exist at

Fig. 5. Typical image pairs and their mean recall and precision values ofKNN-TAR with different values of σ.

all. The value of α has a slight effect on the error. Therefore, itis set to be 5 to pick out the real outliers without removing anycorrect matching pairs.

B. Performance Analysis

In order to evaluate the performance of the proposed method,a comparison between KNN-TAR, GTM, and RSOC is per-formed. GTM is a simple point-matching method which hasbeen proposed in recent years. RSOC is a newly proposedalgorithm which performs well in removing outliers. In thecase of GTM and RSOC, the KNN graph is constructed withthe value of K = 4. The images used in the experiment arearrayed on the order of outlier percentage. The SIFT featuresare extracted from the image pairs, and the initial matchingpairs are obtained by comparing the distance of the closestneighbor with that of the second closest neighbor.

The outlier percentage of the 20 image pairs are shown inFig. 6(a). The performance of the four algorithms in termsof recall, precision, and RMSE are shown in Fig. 6(b)–(d).As shown in the figure, KNN-TAR can successfully removethe outliers from image pairs with large affine transformation,similar patterns, and low overlapping areas. It can be observedthat the recall value of KNN-TAR is the best among the threealgorithms, which is nearly 1. The recall value of RSOC islower than that of KNN-TAR but better than that of GTM. Theprecision and RMSE values of KNN-TAR are the best whencompared with those of the other two algorithms. GTM doesnot performs very well in terms of the RMSE value; it canbe seen that the RMSE values of image pair numbers 4, 5, 8,and 16 are very high. This is because GTM only removes theoutliers according to the local adjacent structure without theguide of the global information; therefore, the matching resultsare not steady. The precision value of RSOC is close to thatof KNN-TAR, while the RMSE value of RSOC is larger thanthat of KNN-TAR. This is because outliers which have the sameKNN cannot be selected as candidate outliers by RSOC, and thenumber of such outliers is so few that the global transformationerror is less than the threshold. As a result, these outliers remainin the matching result of RSOC. On the contrary, these kindsof outliers can be found and removed by the outlier removalalgorithm based on KNN-TAR.

ZHANG et al.: ROBUST POINT-MATCHING ALGORITHM FOR REMOTE SENSING IMAGE REGISTRATION 473

Fig. 6. Performance of three algorithms on 20 image pairs. (a) Outlier percentage. (b) Recall. (c) Precision. (d) RMSE.

TABLE IRUNNING TIMES OF RSOC AND KNN-TAR

C. Time Complexity Analysis

In the proposed algorithm, the distance order matrix betweeneach matching pair is constructed at first; the time used isO(N3). In each iteration, O(KN) is used to compute theKNN-TAR descriptor and select the candidate outliers. For Nc

candidate outliers, O(N2c ) is required to sort the candidate

outliers according to the value of d(ci, θ). Calculating thetransform error changes of one candidate needs O(N). In theworst case, i.e., Nc = N , only the candidate with the maximalvalue of d(ci, θ) is confirmed to be the real outlier by the secondcriterion. The computation time complexity of the proposedalgorithm is as follows:

TKNN-TAR =O(N3) +N ·(O(KN) +O(N2)

+O(N) +O(N))

=O(N3).

An experiment is performed to compare the running times ofRSOC and KNN-TAR. The experiment is executed on an IntelXeon CPU, 2.53-GHz computer with 12-GB RAM in a Matlabenvironment. As shown in Table I, KNN-TAR runs much fasterthan RSOC. This is because the process of calculating thetransformation error of each candidate outlier is the most time-consuming. In each iteration, the transformation error changesof all the candidate outliers are calculated by RSOC, while inKNN-TAR, the calculation will be ended when the first outlierremoval criterion is not satisfied.

Through the experiment discussed in this section, the novelalgorithm proposed in this letter has been proven to be accurateand robust.

IV. CONCLUSION

In this letter, a point-matching algorithm based on KNN-TAR has been introduced to improve the accuracy of image

registration. The experimental results show that the point-matching algorithm performs well for images with large affinetransformation, similar patterns, or low overlapping areas evenwhen a high percentage of outliers exists. Compared with otheralgorithms, the outlier removal algorithm based on KNN-TARperforms better, with higher recall and precision values andlower RMSE. In conclusion, the novel algorithm proposed inthis letter is simple, robust, and efficient in image registration.

ACKNOWLEDGMENT

The authors would like to thank the anonymous reviewersfor their helpful suggestions, the Signal and Image ProcessingInstitute, University of Southern California, and the VisionResearch Laboratory, University of California at Santa Barbara,for the test data, and Z. Liu for providing the source codeof RSOC.

REFERENCES

[1] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.

[2] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigmfor model fitting with applications to image analysis and automated car-tography,” Commun. ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981.

[3] N. Alajlan, I. El Rube, M. S. Kamel, and G. Freeman, “Shape retrievalusing triangle-area representation and dynamic space warping,” PatternRecognit., vol. 40, no. 7, pp. 1911–1920, Jul. 2007.

[4] B. Li and H. Ye, “RSCJ: Robust sample consensus judging algorithmfor remote sensing image registration,” IEEE Geosci. Remote Sens. Lett.,vol. 9, no. 4, pp. 574–578, Jul. 2012.

[5] P. J. Besl and N. D. McKay, “A method for registration of 3-D shapes,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, no. 2, pp. 239–256,Feb. 1992.

[6] Y. Zheng and D. Doermann, “Robust point matching for nonrigid shapesby preserving local neighborhood structures,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 28, no. 4, pp. 643–649, Apr. 2006.

[7] W. Aguilar, Y. Frauel, F. Escolano, M. E. Martinez-Perez,A. Espinosa-Romero, and M. A. Lozano, “A robust graph transformationmatching for non-rigid registration,” Image Vis. Comput., vol. 27, no. 7,pp. 897–910, Jun. 2009.

[8] Z. Liu, J. An, and Y. Jing, “A simple and robust feature point matchingalgorithm based on restricted spatial order constraints for aerial imageregistration,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 2, pp. 514–527, Feb. 2012.

[9] Z. L. Song and J. Zhang, “Remote sensing image registration basedon retrofitted SURF algorithm and trajectories generated from Lissajousfigures,” IEEE Geosci. Remote Sens. Lett., vol. 7, no. 3, pp. 491–495,Jul. 2010.

[10] G. J. Burghouts and J. M. Geusebroek, “Performance evaluation of lo-cal colour invariants,” Comput. Vis. Image Understand., vol. 113, no. 1,pp. 48–62, Jan. 2009.

[11] K. Mikolajczyk and C. Schmid, “A performance evaluation of localdescriptors,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10,pp. 1615–1630, Oct. 2005.