IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 10, NO. 6, NOVEMBER 2013 1429

Bi-SOGC: A Graph Matching Approach Based onBilateral KNN Spatial Orders Around GeometricCenters for Remote Sensing Image Registration

Ming Zhao, Bowen An, Yongpeng Wu, Member, IEEE, and Changqing Lin

Abstract—In this letter, Bilateral K Nearest Neighbors SpatialOrders around Geometric Centers (Bi-SOGC) is presented tomatch feature points for remote sensing images with large affinetransformation, similar patterns or multispectral images. In Bi-SOGC, both the bilateral adjacent relations and the spatial angu-lar orders are considered. Bilateral K Nearest Neighbors (BiKNN)descriptors are proposed to describe the adjacent information.The vertices with maximum BiKNN difference are deemed ascandidate outliers. The invariant spatial angular orders for affinetransformation are used to deal with outliers in pseudo isomorphicstructures, geometric centers are taken as the reference points.To increase the correct matching points and eliminate stubbornoutliers, a recovery strategy utilizes the addition of fresh inliersto break down the stabilized pseudo graphs of the residual sets.Experimental results demonstrate the superior performance ofthis algorithm under various conditions for remote sensing images.

Index Terms—Graph matching, image registration, remotesensing images, spatial orders.

I. INTRODUCTION

IMAGE registration is a crucial preprocessing technologyfor image analysis, which has been widely applied to re-

mote sensing, computer vision and pattern recognition [1], [2].Feature matching in registration process is a challenging stepto determine reliable corresponding relationships of detectedfeatures between the images to be registered.

For remote sensing images, feature matching is interferedwith various factors, such as the large affine transformationand similar patterns caused by different or large field of views.Another intractable issue is the multispectral image registra-tion, of which the gray-level values at the same areas differfrom each other. Compared by Mikolajzyk [3], Scale InvariantFeature Transform (SIFT) [4] is the best among intensity-baseddescriptors for local interest regions, which is invariant to image

Manuscript received December 11, 2012; revised March 6, 2013; acceptedApril 14, 2013. Date of publication June 17, 2013; date of current versionOctober 10, 2013. This work was supported in part by the National NaturalScience Foundation of China under Grants 61302132 and 61171126, the In-novation Program of Shanghai Municipal Education Commission under Grant11ZZ142, and the Shanghai Municipal Natural Science Foundation under Grant11ZR1415200.

M. Zhao is with the Department of Logistics Engineering, Shanghai Mar-itime University, Shanghai 201306, China (e-mail: mingzhao@shmtu.edu.cn).

B. An is with the Department of Information Engineering, Shanghai Mar-itime University, Shanghai 201306, China (e-mail: bwan@shmtu.edu.cn).

Y. Wu is with the National Mobile Communications Research Laboratory,Southeast University, Nanjing 210096, China (e-mail: ypwu@seu.edu.cn).

C. Lin is with Shanghai Institute of Technical Physics, Chinese Academy ofSciences, Shanghai 200083, China (e-mail: hellososo2009@163.com).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/LGRS.2013.2259612

scaling and rotations. Nevertheless, SIFT is partially invariantto intensity change and shear deformations [5]. Therefore,only relying upon the image intensity in feature matching isinsufficient [6], [7], and spatial relations need to be considered.A classic spatial-based approach called Random Sample Con-sensus (RANSAC) estimates transformation parameters frominitial matching sets, but it cannot deal with large proportionof outliers. Aguilar et al. introduced a simple method namedGraph Transformation Matching (GTM) [8] based on matchingK Nearest Neighbor (KNN) graphs with limitations of distancesto remove outliers. However, it may not be adequate to dealwith those outliers with the same local neighbor structurescalled pseudo isomorphic structures. Besides, inliers with dif-ferent neighbor structures might be removed arbitrarily [9].Liu et al. proposed Restricted Spatial Orders Constraints(RSOC) algorithm [10], which integrated the two-way spa-tial order constraints and the transformation error restrictions.However, the convergence rates and accuracy depend on trans-formation models and the initial parameter settings. Also, thecyclic string matching for spatial orders is very time consuming[11]. Izadi et al. proposed Weighted Graph TransformationMatching (WGTM) algorithm [9]. Utilizing the angular dis-tances between edges that connect a feature point to its KNNas the weight, WGTM can only deal with pseudo isomorphicstructures to a certain extent. This arises because angular dis-tance is only invariant with respect to scales and rotations,and shear deformations are not considered in that case.

In this letter, we propose a so called Bilateral KNN SpatialOrders around Geometric Centers (Bi-SOGC) graph matchingalgorithm, and Bilateral K Nearest Neighbors (BiKNN) andspatial orders are considered. First, BiKNN is presented asthe bilateral graph descriptor. Then, the spatial orders aroundgeometric centers (SOGC) are given, which are invariant torotations, scaling and shear deformations. Point matching isformulated as comparison with angular orders between adjacentpoint pairs. In case of the inliers surrounded by outliers as theirKNN points and some inliers with cyclic adjacent orders, arecovering strategy is designed to bring them back. Likewise,with the addition of recovered inliers, the stubborn outliers inthe residual sets might be deleted. Thus, the performance offeature point matching is significantly improved.

II. BILATERAL KNN SPATIAL ORDERS AROUND

GEOMETRIC CENTERS (BI-SOGC)

The input of feature matching algorithms is the one-to-onecorrespondence between a reference image and an image tobe registered. Since the aforementioned ambiguity in the initial

1545-598X © 2013 IEEE

1430 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 10, NO. 6, NOVEMBER 2013

feature point matching based on intensity, the BiKNN descrip-tor is proposed to describe the adjacent relationships with thebilateral KNN. Combined with identical BiKNN, SOGC is pre-sented to constraint the approximate angular position orders ofbilateral adjacent points. Then, a recovery strategy is designedto recover the inliers which have been removed arbitrarily.

A. BiKNN Descriptor

Suppose that two sets of corresponding points are V = {vi}and V ′ = {v′i}. In BiKNN, bilateral KNN graphs Gp = (Vp,−→E p) and G′

p = (V ′p,−→E

′p) are established. Likewise, N ×

N bilateral KNN matrices, i.e., forward adjacent matri-ces (FKNN,FKNN′) and backward adjacent matrices(BKNN,BKNN′) are constructed as follows (we take V ={vi} as an example):

1) A directed edge 〈vi �→ vj〉 ∈−→E p starts from i to j exists

when vj is one of the K closest neighbors of vi;2) FKNN[i, j] = 1 when 〈vi �→ vj〉 ∈

−→E p and

BKNN[i, j] = 1 when 〈vi �→ vj〉 ∈−→E p; Otherwise,

FKNN[i, j] = 0, BKNN[i, j] = 0.Forward and backward KNN point sets are established as

Fknn(i) ={vj |〈vi �→ vj〉 ∈

−→E p, ∀j ∈ Vp

}(1)

Bknn(i) ={vj |〈vj �→ vi〉 ∈

−→E p, ∀j ∈ Vp

}. (2)

Based on 1) and 2), structure disparity is estimated by thetwo adjacency difference matrices ΔFKNN = |FKNN−BKNN′| and ΔBKNN = |BKNN−BKNN′|. The can-didate outliers mainly rely on the differences of forwardand backward connections between two corresponding graphs.Thus, the candidate outliers joutlier could be selected with thefollowing structural criterion:

joutlier=argmax∀j∈Vp,V ′

p

N∑i=1

(ΔFKNN[i, j]+ΔBKNN[i, j]). (3)

Similar to GTM in [6], all references to the outlier ver-tices (vjoutlier , v′joutlier) are removed. A new iteration beginswith the decrement of residual vertices until ΔFKNN[i, j] =ΔBKNN[i, j] = 0, ∀i, j. At this stage, there is no differencebetween the two graph structures estimated by FKNN andBKNN.

Fig. 1 shows an example of the graph matching processwith K = 4 for remote sensing images from different views. Itreaches the identical graphs in the final iteration 22. Nonethe-less, the BiKNN descriptor is a non-rigid constraint for match-ing, since only adjacent point sets are considered. BiKNN willfail in the cases that outliers with pseudo isomorphic adjacentstructures or inliers have different adjacent structures.

B. Spatial Orders Around Geometric Centers

Since the angular spatial order is invariant to affine trans-formation, the relative spatial orders of BiKNN points around(vi, v

′i) should be the same. Spatial Orders around Geometric

Centers (SOGC) are designed to remove the residual outliers inthe pseudo isomorphic graphs.

The identical BiKNN graphs for (VBiKNN , V ′BiKNN ) have

been obtained by BiKNN. The adjacent sequences of corre-

Fig. 1. Iterative process of BiKNN. (a) G(VR) of iteration 0. (b) G(VR) ofiteration 10. (c) G(VR) of iteration 22. (d) G′(VT ) of iteration 0. (e) G′(VT )of iteration 10. (f) G′(VT ) of iteration 22.

Fig. 2. Spatial orders for BiKNN pseudo isomorphic graph structures (K =3). (a) The spatial graph structure of V . (b) The spatial graph structure of V ′.

sponding points may start with different adjacent points due torotation or shear transformation, even if all points are matched.Therefore, there will be cyclic orders for rotated or shearedgraphs to disturb the matching points. The geometric centers(Op, O

′p) defined below are chosen as reference points for

corresponding graphs⎧⎪⎪⎨⎪⎪⎩

(xOp

, yOp

)=

(1N

N∑i=1

xi,1N

N∑i=1

yi

)(x′Op

, y′Op

)=

(1N

N∑i=1

x′i,

1N

N∑i=1

y′i

).

(4)

The accuracy of (Op, O′p) representing matching centers

depends on the proportion of outliers in (VBiKNN , V ′BiKNN ).

The more inliers there are, the more likely (Op, O′p) are

matching points. Polar coordinate systems are constructed foreach vertex. The vectors from inspected points (vi, v

′i) to

(Op, O′p) are denoted as the polar axes for (vi, v′i), i.e., −→ox =

〈vi �→ Op〉, −→ox′ = 〈vi �→ O′p〉. The forward and backward spa-

tial orders (SFknn(i), SBknn(i)) for (Fknn(i), Bknn(i) aresorted by the polar angles between the directive edges ofBiKNN (〈vi �→ Fknn(i)〉, 〈vi �→ Bknn(i)〉) with −→ox, respec-tively. (SFknn′(i), SBknn′(i)) are obtained in the same way.

Fig. 2(a) and (b) demonstrate the rotated BiKNN graphs ofcorresponding sets, which are pseudo isomorphic structuresafter BiKNN iterations because of the outliers (v5, v

′5). The

ZHAO et al.: BI-SOGC: AN APPROACH BASED ON BILATERAL KNN SPATIAL ORDERS AROUND GEOMETRIC CENTERS 1431

bilateral spatial orders for outliers (v5, v′5) are different, e.g.,

SFknn(5) = {v3, v6, v4}, SFknn′(5) = {v′4, v′3, v′6} andSBknn′(5) = {v3, v6, v4}, SBknn′(5) = {v′4, v′3, v′6}.

In SOGC, each vertex with identical adjacent spatial orderscan be reserved as inliers. Otherwise, the vertex pairs with dis-orderly or cyclic adjacent orders should be treated as candidateoutliers and removed in the iterative process

joutlier={SFknn(j) =SFknn′(j)‖SVknn(j) =SBknn′(j)} .(5)

The BiKNNs are updated with the decreasing vertices ineach new round. The SOGC iterates until BiKNN spatial ordersare all the same. Cyclic spatial orders for several vertices areambiguous in SOGC. Incorrect matching geometric centers cancause cyclic spatial orders for several inliers, not only outliers.Outliers are intolerant, and the matching accuracy of (Op, O

′p)

mainly depends on the proportion of outliers. Therefore, thevertices with cyclic adjacent spatial orders are removed to makea compromise for faster convergence speed and precisions.

C. Recovery Strategy

A certain amount of inliers are removed in previous steps,especially the vertices surrounded by a large proportion ofoutliers. Likewise, there might be some stubborn outliers in theresidual vertex sets. They cannot be selected as outliers withoutany addition of fresh vertices.

The residual vertices with each candidate outlier removed inBi-SOGC are constructed as new checking sets. In each time,there is only one candidate outlier to be checked. Those inlierswith exactly the same BiKNN and SOGC structures com-pose the recovery sets (Vrecover, V

′recover), e.g., ΔFKNN =

ΔBKNN = 0, SFknn = SFknn′, SBknn = SBknn′ forall vertices in checking sets.

The recovery criterion is only on account of one incrementfor residual sets. When all points in (Vrecover, V

′recover) are re-

covered into residual sets, the BiKNN and spatial orders mightno longer be the same. That is why a new round of Bi-SOGCis put forward for the combination of (Vresidual, V

′residual) and

(Vrecover, V′recover) until there are no points need to be recovered.

Fig. 3 depicts the matching process with Bi-SOGC for mul-tispectral images from Landsat TM band 3 and 5. Differentadjacent structures with SOGC are marked out in cyan. Theaxes of outliers are labeled by red dotted lines. With threeiterations, the graph structures and spatial orders are identicalfor each corresponding vertex as shown in Fig. 3(i).

III. EXPERIMENTS AND ANALYSIS

To preclude other impacts on the performance analysis, SIFTfeature points are extracted by differences between Gaussian fil-ters for all demonstrations, and a 128-element vector is assignedfor each feature point. The initial matching correspondence isobtained by comparing the distance of the closest SIFT vectorswith second closest ones.

A. Evaluation Criterion

Recall is the number of correctly matched features with re-spect to the number of initial matched features between two im-ages, e.g., Recall = CorrectMatches/TotalMatches. Thus,

Fig. 3. Iterative process of Bi-SOGC. (a)–(b) BiKNN in iteration 1.(c) SOGC in iteration 1. (d)–(e) BiKNN in iteration 2. (f) SOGC in iteration 2.(g)–(h) BiKNN in iteration 3. (i) SOGC in iteration 3.

it is influenced by feature extractions. The correctly match-ing number relative to the total number of points is repre-sented by Precision, e.g., Precision = CorrectMatches/FalseMatches. Precision focuses on the matching results.

B. Experimental Results

Eighty real challenging image pairs are chosen to evaluatethe performance of GTM, RANSAC, BiKNN, and Bi-SOGC,which are divided into four groups: 1) Test 1: 20 single bandimages from different views; 2) Test 2: 20 single band imageswith rotations; 3) Test 3: 20 single band images with sheardeformations; 4) Test 4: 20 multispectral images. Fig. 4 demon-strates four samples of matching results in aforementionedambiguous situations. Pseudo isomorphic structures are moreprone to occur in GTM and BiKNN, and much more outliersare retained in RANSAC than the others.

Rotation and shear deformations affect the spatial-based ap-proach directly by geometrical relationship. Twenty remotesensing images included in Tests 1–3, respectively from singleband are chosen to compare the performance of four algorithmsbased on KNN with different K, rotations and shear deforma-tions. Inliers are reserved by hand, and outliers are randomlyadded into the initial matching sets in increments of 10% from5% to 95%.

Fig. 5 depicts the statistic results of the three algorithmsbased on KNN with different K. Bi-SOGC is superior to theother two algorithms in terms of recall and precision rates.GTM and BiKNN are more sensitive to K than Bi-SOGC. Thestability of Bi-SOGC comes from the SOGC constraint forresidual outliers and the recovery strategy for inliers removedpreviously.

Fig. 6 demonstrates the statistic results of Test 2 with10◦–30◦ and 60◦–90◦ rotations for GTM, BiKNN, and Bi-SOGC. The recall and precision rates of the four algorithmsare little changed with rotations, compared with those of norotation. It indicates the invariant to rotations of the spatial-based matching algorithms.

1432 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 10, NO. 6, NOVEMBER 2013

Fig. 4. Matching results for Image samples. (a)–(d) initial matching by SIFT. (e)–(h) RANSAC matching. (i)–(l) GTM matching. (m)–(p) BiKNN matching.(q)–(t) Bi-SOGC matching.

Fig. 5. Performance comparison for GTM, BiKNN, Bi-SOGC with K = 4, 10, 15, 20. (a) and (d) Mean recall and precision of GTM with different K. (b) and(e) Mean recall and precision of BiKNN with different K. (c) and (f) Mean recall and precision of Bi-SOGC with different K.

Fig. 6. Performance comparison for Bi-SOGC, BiKNN, GTM, and RANSACunder different rotations. (a) and (d) 0◦ rotation mean recall and precision. (b) and(e) 10◦–30◦ rotation mean recall and precision. (c) and (f) 60◦–90◦ rotation mean recall and precision.

Fig. 7 describes the statistic results of different sheardeformations for four algorithms. Different from the rigidtransformation such as rotations, shear deformations cause theinconformity of relative distance between vertices. The recall

rates are decreased with shear deformations for each algo-rithm, while GTM is affected the most seriously. Likewise, theprecision rates of RANSAC are the lowest of all. That arisesbecause RANSAC is strict on the decision for outliers.

ZHAO et al.: BI-SOGC: AN APPROACH BASED ON BILATERAL KNN SPATIAL ORDERS AROUND GEOMETRIC CENTERS 1433

Fig. 7. Performance comparison for BiSOGC, BiKNN, GTM, and RANSAC under different shear deformations. (a) and (d) No shear mean recall and precision.(b) and (e) H = 0.1, V = 0.1 shear mean recall and precision. (c) and (f) H = 0.2, V = 0.2 shear mean recall and precision.

Fig. 8. Performance comparison of four algorithms for multispectral images with similar patterns. (a) Initial matching for 20 image pairs. (b) Recall. (c) Precision.(d) Execution time.

Multispectral images with similar patterns have indirect im-pact on image registration by mismatches in confusing regions.Images in such conditions from Landsat 5 TM Band 1–7 andLandsat 7 EM+Band 1–8 in Test 4 are chosen for comparisons.The initial correspondence is constructed by SIFT shown inFig. 8(a), which indicates its vulnerability for identical regionswith inconsistent intensities. The high recall rates of RANSACare close to Bi-SOGC, but it degenerates much more seriouslythan other three algorithms with increasing proportion of out-liers in initial sets. Bi-SOGC is superior to those of both GTMand BiKNN in terms of recall and precision rates.

C. Complexity Analysis

The most time consuming process in GTM is computingand sorting distances between pairs of vertices. FKNN andBKNN in BiKNN can be assigned as well as comparing edgedistances in GTM. Thus, the complexity of BiKNN is almostthe same as GTM. The number of initial points and proportionsof outliers determine the iterative times in BiKNN, GTM andBi-SOGC. In Bi-SOGC, the time complexity decomposed foreach step in the first iteration is as follows (n is the initial pointnumber, K is the adjacent size):

1) Creating the distances and sorting for KNN in 2 BiKNN:O(n2 + n2log(n));

2) Sorting the angular spatial orders of Fknn(i) in SOGC:O(2K × n2);

3) Computing the KNN and spatial orders for candidateoutliers in recovery strategy: O(3K × n).

In 2), the time complexity can be reduced by removingvertices once there are any mismatches in adjacent spatialorders. The execution time demonstration for Test 4 is shownin Fig. 8(d).

IV. CONCLUSION

To improve feature matching in remote sensing image reg-istration, which is influenced by large affine transformations,similar pattern and multispectral images, Bi-SOGC has been

proposed. First, BiKNN representing the detailed adjacent re-lationships was constructed for each vertex. The vertices withmaximum disparities in adjacent structures were deemed ascandidate outliers. Then, the angular spatial orders invariant toaffine transformations were sorted by the adjacent polar angles,to deal with pseudo isomorphic structures. Finally, a recoverystrategy was designed to retrieve inliers arbitrarily deleted inthe preceding steps. Simultaneously, stubborn outliers couldbe settled down with the addition of fresh points. Experimentswere carried out with typical remote sensing images. Comparedwith RANSAC, GTM and BiKNN, Bi-SOGC has been provedto be more robust and efficient in matching feature points forremote sensing images.

REFERENCES

[1] B. Zitoza and B. Flusser, “Image registration methods: A survey,” ImageVis. Comput., vol. 21, no. 11, pp. 977–1000, Oct. 2003.

[2] T. S. Caetano and J. J. McAuley, “Learning graph matching,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 31, no. 6, pp. 1048–1058, Jun. 2009.

[3] K. Mikolajzyk and C. Schmid, “A performance evaluation of local de-scriptors,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10,pp. 1615–1630, Oct. 2005.

[4] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.

[5] Z. Yi, C. Zhaiguo, and X. Yang, “Multi-spectral remote image registrationbased on sift,” Electron. Lett., vol. 44, no. 2, pp. 107–108, Jan. 2008.

[6] H. Bunke and B. T. Messmer, “Recent advances in graph matching,” Int.J. Patt. Recog. Artif. Intell., vol. 11, no. 1, pp. 169–203, Feb. 1997.

[7] G. J. Wen, J. J. Lv, and W. X. Yu, “A high-performance feature-matchingmethod for image registration by combining spatial and similarity infor-mation,” IEEE Trans. Geosci. Remote Sens., vol. 46, no. 4, pp. 1206–1277, Apr. 2008.

[8] W. Aguilar, Y. Frauel, F. Escolano, M. E. Martinez-Perez, A. Espinosa-Romero, and M. A. Lozano, “Arobust graph transformation matching fornon-rigid registration,” Image Vis. Comput., vol. 27, no. 7, pp. 897–910,Jun. 2009.

[9] M. Izadi and P. Saeedi, “Robust weighted graph transformation matchingfor rigid and nonrigid image registration,” IEEE Trans. Image Process.,vol. 21, no. 10, pp. 4369–4382, Oct. 2012.

[10] Z. X. Liu, J. B. An, and Y. Jing, “A simple and robust feature pointmatching algorithm based on restricted spatial order constraints for aerialimage registration,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 2,pp. 514–527, Feb. 2012.

[11] X. Dian and S. Khorram, “A feature-based image registration algorithmusing improved chain-code representation combined invariant moments,”IEEETrans.Geosci.RemoteSens., vol.37, no.5, pp.2351–2362,Sep.1999.