Accuracy assessment of high resolution satellite imagery orientation by leave-one-out method

ISPRS Journal of Photogrammetry & Remote Sensing 63 (2008) 427–440www.elsevier.com/locate/isprsjprs

Accuracy assessment of high resolution satellite imagery orientationby leave-one-out method

Maria Antonia Brovellia,∗, Mattia Crespib,1, Francesca Fratarcangelib,1,Francesca Giannoneb,1, Eugenio Realinia,2

a DIIAR, Politecnico di Milano, Polo Regionale di Como, via Valleggio 11 - 22100 Como, Italyb DITS - Area di Geodesia e Geomatica, Universita di Roma “La Sapienza” via Eudossiana, 18 - 00184 Rome, Italy

Received 7 November 2006; received in revised form 15 January 2008; accepted 21 January 2008Available online 9 May 2008

Abstract

Interest in high-resolution satellite imagery (HRSI) is spreading in several application fields, at both scientific and commerciallevels. Fundamental and critical goals for the geometric use of this kind of imagery are their orientation and orthorectification,processes able to georeference the imagery and correct the geometric deformations they undergo during acquisition. In order toexploit the actual potentialities of orthorectified imagery in Geomatics applications, the definition of a methodology to assess thespatial accuracy achievable from oriented imagery is a crucial topic.

In this paper we want to propose a new method for accuracy assessment based on the Leave-One-Out Cross-Validation(LOOCV), a model validation method already applied in different fields such as machine learning, bioinformatics and generally inany other field requiring an evaluation of the performance of a learning algorithm (e.g. in geostatistics), but never applied to HRSIorientation accuracy assessment.

The proposed method exhibits interesting features which are able to overcome the most remarkable drawbacks involved by thecommonly used method (Hold-Out Validation — HOV), based on the partitioning of the known ground points in two sets: the firstis used in the orientation–orthorectification model (GCPs — Ground Control Points) and the second is used to validate the modelitself (CPs — Check Points). In fact the HOV is generally not reliable and it is not applicable when a low number of ground pointsis available.

To test the proposed method we implemented a new routine that performs the LOOCV in the software SISAR, developed by theGeodesy and Geomatics Team at the Sapienza University of Rome to perform the rigorous orientation of HRSI; this routine wastested on some EROS-A and QuickBird images. Moreover, these images were also oriented using the world recognized commercialsoftware OrthoEngine v. 10 (included in the Geomatica suite by PCI), manually performing the LOOCV since only the HOV isimplemented.

The software comparison guaranteed about the overall correctness and good performances of the SISAR model, whereas theresults showed the good features of the LOOCV method.c© 2008 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). Published by Elsevier B.V. All rights reserved.

Keywords: High resolution satellite imagery; Orientation; Accuracy assessment; Leave-one-out cross validation

∗ Corresponding author. Tel.: +39 0313327517; fax: +39 0313327519.E-mail addresses: [email protected] (M.A. Brovelli), [email protected] (M. Crespi), [email protected]

(F. Giannone), [email protected] (E. Realini).1 Tel.: +39 0644585068; fax: +39 0644585515.2 Tel.: +39 0313327517; fax: +39 0313327519.

0924-2716/$ - see front matter c© 2008 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). Published by Elsevier B.V.All rights reserved.doi:10.1016/j.isprsjprs.2008.01.006

http://www.elsevier.com/locate/isprsjprs

mailto:[email protected]




http://dx.doi.org/10.1016/j.isprsjprs.2008.01.006

428 M.A. Brovelli et al. / ISPRS Journal of Photogrammetry & Remote Sensing 63 (2008) 427–440

1. Introduction

High resolution satellite imagery (HRSI) becameavailable in 1999 with the launch of IKONOS, thefirst civil satellite offering a spatial resolution of 1 m.Since then other high resolution satellites have beenlaunched, among which there are EROS-A (1.8 m),QuickBird (0.61 m), Orbview-3 (1 m), EROS-B (0.7m) and Worldview-1 (0.5 m), and others are planned tobe launched in the next few years, as for example Geo-Eye (0.41 m) and Worldview-2 (0.5 m). Scientific andcommercial interests in this kind of remote sensed datahave focused (Jacobsen, 2002; Fraser, 2003; Toutin andCheng, 2002) on their use for cartographic purposes,since they proved to be a suitable alternative to aerialphotogrammetric data for producing orthophotos up to1:10 000–1:5000 scale (Amato et al., 2004) or lower.High resolution remote sensing offers some advantagesover traditional photogrammetric techniques. It enablesusers to acquire easily the same portion of terrain atregular intervals depending only on the satellite revisittime, which is useful to monitor natural or technologicalphenomena evolving in time, e.g. urban growing,natural disasters. It also allows us to obtain images ofgeographic areas where it can be difficult to arrangephotogrammetric flights (e.g. developing countries).Although HRSI still cannot replace aerial photos, whichprovide resolutions as high as few centimeters, some ofthe new satellites that are going to be launched in thenext couple of years, offering resolutions of 0.4–0.5 m,are expected to narrow the gap between satellite imagesand aerial photos.

The metric use of HRSI for map making (orupdating) usually requires them to be oriented in areference system and orthorectified. Orthorectification(Section 2) is a process aimed to correct the geometricdistortions these images undergo during acquisition, inorder to be able to use them effectively for measurementpurposes and geomatics applications. Therefore it iscrucial to assess the accuracy achievable from the finalproduct, which strictly depends on the model chosen forthe orientation, on the original image characteristics andon the quality of the known ground points coordinates,which are generally used to guarantee an higherorientation reliability. Currently, such assessments areperformed through the validation technique known asHold-Out Validation (HOV) method (Section 3).

In the present work we propose a new techniqueto perform spatial accuracy assessments on orientedHRSI, that is the use of the LOOCV method(Section 3), a model validation method already usedin geostatistics and model theory but never applied

to HRSI orientation accuracy assessment. This newmethod exhibits interesting features, which are able toovercome the most serious drawback of the commonlyused HOV method: HOV is generally not reliable and itis not applicable when a low number of ground points isavailable.

Various experiments (Section 5) have been carriedout to test the proposed method as an alternative tothe classical procedure, and to compare the resultsobtained by the original software SISAR (Section 4)to those supplied by the world recognized commercialsoftware OrthoEngine v. 10 (PCI Geomatica) in orderto be guaranteed about the overall correctness and goodperformances of the SISAR model.

2. Orientation and orthorectification

As previously pointed out, satellite images undergogeometric distortions during acquisition. Distortionssources can be related to two general categories(Toutin et al., 2003): the acquisition system, whichincludes the platform orientation and movement andthe imaging sensor optical-geometric characteristics,and the observed object, which takes into accountatmosphere refraction and terrain morphology.

Overall, the orthorectification process may besubdivided into two steps: the orientation and thefollowing orthorectification. In the former, an imageis georeferenced in a defined reference frame (usuallya WGS84 realization) and an analytical relationshipbetween the image and Earth coordinates is established;at this step the distortions due to the acquisition systemand to the atmospheric refraction are corrected. Inthe latter the distortion due to terrain morphology iscorrected by means of a suitable digital elevation orsurface model (DEM/DSM), starting from the orientedimage.

The most reliable models to perform the orientation(and then, the orthorectification) can be classified intwo categories: physically based models (also calledrigorous models) and models based on purely analyticalfunctions.

A rigorous orientation model describes the entireacquisition process in all of its fundamental physical-geometric aspects, including satellite position, sensorattitude and characteristics, atmosphere refractioneffect, and a possible final cartographic transformation(Toutin, 2004). Such models are based on platform-specific data provided by the image vendor. Thosedata are the satellite orbital parameters, the attitudeangles and the sensor interior orientation parameters.The provided values must be refined by estimating their

M.A. Brovelli et al. / ISPRS Journal of Photogrammetry & Remote Sensing 63 (2008) 427–440 429

corrections using a suitable number of GCPs (Chen andTeo, 2002).

The whole method is based on establishing arelation between the image and the terrain coordinateswith traditional photogrammetric collinearity equations.Nevertheless it has to be noted that, unlike traditionalphotogrammetry, where there is a single camera center,in satellite photogrammetry there are multiple cameracenters, as the satellite moves during the imageacquisition. In addition, in order to relate the imageto the ground coordinates, expressed in an EarthCentered–Earth Fixed (ECEF) reference frame, a set ofrotation matrices has to be used. These matrices includethose needed to shift between sensor, platform, orbitaland Earth Centered Inertial (ECI) coordinate systems(Westin, 1990), while the transformation between ECIand ECEF coordinate systems must take into accountprecession, nutation, polar motion and Earth rotationmatrices (Kaula, 1966).

Another important aspect of photogrammetry fromspace is that the positions of a generic image pointand its correspondent object point and camera centerdo not lie on a line because of atmosphere refractioneffect; therefore a correction to the apparent positionof each point must be applied; a possible procedure toaccomplish this is detailed in Noerdlinger (1999).

A model based on purely analytical functions is in-dependent from specific platform/sensor characteristicsand acquisition geometry. In this case, usually ratio-nal polynomial functions (RPFs) with known coeffi-cients (Rational Polynomial Coefficients — RPCs) areused (Di et al., 2003; Fraser and Hanley, 2005; Fraseret al., 2006); these are at most third order functions,since higher orders do not substantially improve the re-sults. The RPCs are computed by the companies thatmanage the satellites using their proprietary rigorousmodels. Therefore, RPFs with RPCs are a sort of a poly-nomial representation of the rigorous models for eachimage. The RPCs are supplied into the imagery meta-data, but not for all the satellites: EROS-A, EROS-Band, in some cases, also IKONOS are not provided with(Noguchi et al., 2004). Moreover, since RPCs are rou-tinely supplied for standard format images, they are notavailable in case of particularly extended ones, coveringwide areas (e.g. the Salerno image here considered is47 × 18 km2).

In both cases, the orientation accuracy assessment,and also its reliability, must be based on knownground points, which may also act as ground controlpoints (GCPs) to estimate the unknown parameters ofrigorous models or to refine the orientation based on theRPCs through zero (translation) or first order (affine)

transformations (Fraser et al., 2006; Zhang, 2005). Thecrucial question is: what is the best way to use theknown ground points, which may be a few since theircollection at the accuracy level suitable to exploit thehighest HRSI orientation accuracy may be difficult,expensive and time consuming?

3. Accuracy assessment strategies

3.1. Hold-Out Validation

Currently, the most used method to assess spatialaccuracy of oriented HRSI is the Hold-Out Valida-tion (HOV), also known as test sample estimation. Ac-cording to it, the data set (known ground points) ispartitioned in two subsets: the first one used into theorientation–orthorectification model (GCPs — GroundControl Points) and the second one to validate the modelitself (CPs — Check Points). The only restriction onsuch selection is to have both sets sufficiently well-distributed on the whole image; apart from this con-sideration, the selection should be random. Once themodel is trained, accuracy is usually evaluated as theRoot Mean Squared Error (RMSE) of residuals betweenimagery derived coordinates with respect to CPs coor-dinates.

This method has the advantage of being simple andeasy to compute, but it also has some drawbacks, as itis generally not reliable and it is not applicable when alow number of ground points is available. First of all,once the two sets are selected, accuracy estimate is notreliable since it is strictly dependent on the points usedas CPs; if outliers or poor quality points are includedin the CPs set, accuracy estimate is biased. In addition,when a low number of ground points is available,almost all of them are used as GCPs and very few CPsremain, so that RMSE may be computed on a poor (notsignificant) sample. In these cases accuracy assessmentwith the usual procedure is essentially lost. In addition,this method displays a low efficiency, making a poor useof the available information, as a large part of it must becollected and just used for validation purposes.

3.2. Leave-One-Out Cross-Validation

The proposed alternative to the HOV to performaccuracy assessments of orthorectified HRSI is theLeave-One-Out Cross-Validation (LOOCV) method.The LOOCV is a statistical estimation techniquecurrently applied in different fields such as machinelearning (Elisseeff and Pontil, 2002), bioinformatics(Simon et al., 2003) and generally in any other


field requiring an evaluation of the performance ofa learning algorithm (e.g. in geostatistics) (Kohavi,1995). It is a special case of the k-fold cross-validationmethod (Stone, 1974; Geisser, 1975), which involvesthe partitioning of the original data set in k subsetsof (approximately) equal size. The model is trained ktimes, using each subset in turn as test set, with theremaining subsets being the training set. The overallaccuracy can be obtained averaging the accuracy valuescomputed on each subset. The LOOCV is a k-fold cross-validation computed with k = n, where n is the size ofthe original data set. Each test set is therefore of size 1,which implies that the model is trained n times.

Our proposal consists of applying the LOOCV as aneffective accuracy evaluation method for the orientationof HRSI, being particularly useful when a low numberof ground points is available.

This method applied to HRSI orientation involves theiterative application of the orientation model using allthe known ground points as GCPs except one, differentin each iteration, used as a CP. In every iterationthe residual between imagery derived coordinates withrespect to CP coordinates (the model prediction erroron CP coordinates) is calculated. The overall spatialaccuracy achievable from the oriented image may beestimated by calculating the usual RMSE or, better, arobust accuracy index of the prediction errors (ei ) on allthe iterations (i = 1, n); here mAD (median AbsoluteDeviation)

mAD = median|ei | i = 1, n (1)

is considered as a robust index, but also MAD (MeanAbsolute Deviation) might be used as well.

In this way we solve both drawbacks of the classicalprocedure mentioned above: it is a reliable and robustmethod, not dependent on a particular set of CPs oron outliers, and it allows us to use each known groundpoint both as a GCP and as a CP, capitalising allthe available ground information. Obviously, this isparticularly relevant when the ground point number iskept as low as possible due to budget and/or logisticconstraints.

Both HOV and LOOCV may be obviously appliedboth with a rigorous and with a RPC-based (withpossible zero or first order correction) orientationmodel. In this context, due to their wider chances ofapplication, rigorous models are only considered.

To test the proposed method we modified thesoftware SISAR, developed by the Geodesy andGeomatics Team at the Sapienza University of Rome(Italy) to perform rigorous orientation of EROS-A,QuickBird and IKONOS imagery, integrating it with

a module suitable to carry out iteratively the corealgorithm with point configurations required to performthe LOOCV. Moreover, in order to be more confident ofthe obtained results, SISAR was compared to the worldrecognized commercial software OrthoEngine v.10(PCI Geomatica), which required manual iterations forthe LOOCV method, since only HOV is implementedwithin it.

4. Orientation software: SISAR rigorous model

In this paragraph only the characteristics of theSISAR software will be outlined, since those ofthe other software used for this work, i.e. PCIOrthoEngine, are widely known, being one of theleading photogrammetric software packages at worldlevel.

Since 2003, the research group of the Geodesyand Geomatics Team at the Sapienza University ofRome has developed original rigorous models designedfor the orientation of HRSI acquired by pushbroomsensors carried on satellite platforms with asynchronousacquisition mode, like EROS-A, QuickBird (Baiocchiet al., 2004; Crespi et al., 2006) and IKONOS.

From now on, we only consider and discuss insome details the model suitable for EROS-A andQuickBird basic imagery, which is based on a standardphotogrammetric approach describing the physical-geometrical imagery acquisition.

Of course, in this case, it has to be considered thatan image stemming from a pushbroom sensor is formedby many (from thousands to tens of thousands) lines,each characterized by a proper position (projectioncenter) and attitude. All the acquisition positions arerelated by the orbital dynamics. Therefore, the rigorousmodel implemented in SISAR is based on collinearityequations, with the reconstruction of the orbital segmentduring the image acquisition through the knowledge ofthe acquisition mode; the collinearity equations includethe sensor optical parameters (internal orientation andself-calibration), the sensor attitude parameters and thesatellite position:

• the internal orientation and self-calibration parame-ters include the perspective center position, a rotationin the CCD array plane and a 2nd order lens distor-tion (5 parameters)

• the sensor attitude is supposed to be represented by aknown time-dependent term plus a 2nd order time-dependent polynomial, one for each attitude angle(ϕ, ϑ,ψ) (9 parameters)


• the satellite position is described through the sixKeplerian orbital parameters corresponding to theorbital segment during the image acquisition plus thetime at the perigee passage (7 parameters)

• atmospheric refraction is also taken into account bya general remote sensing model (Noerdlinger, 1999);note that atmospheric refraction is relevant underlarge off-nadir angles (0.5 m at 10◦, 7 m at 50◦).

In order to relate the image to the groundcoordinates, expressed in an Earth Centered–EarthFixed (ECEF) reference frame (usually a realization“close to” WGS84, e.g. ITRF2005 or IGS05), a setof rotation matrices has to be used. These matricesinclude those needed to shift between sensor, platform,orbital and Earth Centered Inertial (ECI) coordinatesystems, while the transformation between ECI andECEF coordinate systems must take into accountprecession, nutation, polar motion and Earth rotationmatrices (Kaula, 1966).

In our case, the collinearity equations may beconveniently expressed in the ECI system in vector form

R1 |X t I −X SI | −

(dpix

f[J − int(J )− 0.5 − J0

− k(I − I0)]

)R3 |X t I −X SI | = 0

R2 |X t I −X SI | +

(dpix

f[k(J − int(J )− 0.5 − J0)

− (I − I0)+ d1(I − I0)+ d2(I − I0)2]

)× R3 |X t I −X SI | = 0

(2)

where: (X t I , Yt I , Z t I ) are the ECI coordinates of theground point and (I, J ) are its image coordinates (originis in the image upper left corner, I is the column andJ is the row, with I increasing toward the right and Jincreasing in the downward direction); (Xs I , Ys I , Zs I )

are the ECI coordinates of the satellite; (R1, R2, R3)

are the rows of the rotation matrix from the ECI tothe sensor system, expressed on the basis of the sensorattitude and the satellite position parameters; (I0, J0)

are the perspective center position; k is the rotationin the CCD array plane; dpix is the (square) pixeldimension; f is the focal distance; (d1, d2) are the lensdistortion parameters.

Approximate values of all these parameters can becomputed thanks to the information contained in themetadata file delivered with each image, so that thecollinearity equations can be linearized and solved ina least squares sense in order to correct the approximate

values, provided a suitable number of Ground ControlPoints (GCPs) is available.

Nevertheless, since the orbital arc related to eachimage acquisition is usually extremely short (fewhundreds of kilometers) if compared to the whole orbitlength (tens of thousandths of kilometers), in most casessome Keplerian parameters are not estimable at all andthe others are extremely correlated both among themand with sensor attitude, internal orientation and self-calibration parameters. Moreover, the 2nd order time-dependent polynomial for the sensor attitude is notalways required, depending on the (mean value of the)off-nadir angle and its first derivative.

In such a situation, the actual number of estimableparameters may change from image to image, so that, inorder to avoid instability due to high correlations amongsome parameters leading to design matrix pseudo-singularity, Singular Value Decomposition (SVD) andQR decomposition are employed to evaluate theactual rank of the design matrix, to select theestimable parameters and finally to solve the linearizedcollinearity equations system in the least squaressense (Golub and Van Loan, 1993; Strang and Borre,1997). The threshold used to exclude a parametercorrection from the estimation is based on the statisticalsignificance of its impact on the estimated variance ofunit weight σ0 (Press et al., 1992).

As usual, the solution is obtained iteratively due tonon linearity; the iterative procedure is stopped whenthe estimated variance of unit weight σ0 reaches aminimum.

Moreover, the statistical significance of the estimableparameter corrections are checked by a Fisher F-test soto avoid over parameterization; in case of no statisticallysignificant parameter correction, they are removed andthe estimation process is repeated until all correctionsare significant.

As regards the stochastic model, a simple diagonalcofactor matrix for the observations (I, J ; X t , Yt , Z t ) isassumed; standard deviations of the image coordinates(I, J ) are set equal to 0.5 pixel, considering thatmanual collimation tests carried out independentlyby different operators showed an image coordinatesaccuracy ranging from 1/3 to 1/2 pixel; for the GCPcoordinates standard deviations are usually set equal tomean values obtained during their surveys.

5. Results

The experimental tests were carried out on twoEROS-A Basic and two QuickBird Basic images; their


Table 1Processed images main features

Main features ITA1-e1038452 ITA1-e1090724 04JAN06093307-P1BS-000000130187 01 P002

SALERNO

Location Rome Rome Augusta (Sicily) SalernoMin (lat, lon) 41.8452, 12.4337 41.8355, 12.3916 37.0813, 15.0636 40.4053, 14.8189Max (lat, lon) 41.9374, 12.5826 41.9494, 12.5989 37.2702, 15.2812 40.8449, 15.0357Satellite EROS-A EROS-A QuickBird QuickBirdDate 2001-08-14 2002-07-22 2004-06-06 2005-07-17γ start 9.1◦ 31.0◦ 28.2◦ (mean) 20.0◦ (mean)γ end 9.4◦ 40.1◦ – –Area 10 × 12 km2 18 × 12 km2 20 × 20 km2 47 × 18 km2

(“triple image”)Mean GSD 1.8 m 2.6 m 0.75 m 0.67 mNumber of availableground points

49 49 39 55

GSD — Ground sample distance, γ — off-nadir angle.

technical details, extracted from the attached metadatafile, are listed in Table 1.

All the ground points were carefully GNSS (GPS andGlonass) surveyed with horizontal and vertical accuracyranging from 10 to 20 cm; in this respect, no outliershave to be suspected in ground points coordinates.

The first step of the investigation was devoted toassess the accuracy and the minimum number of GCPsto achieve it for each image. In fact, it was provedin many situations that, when GCPs number increases,the image accuracy increases reaching a maximum; nofurther enhancement may be got even if other GCPsare added. In this step we applied the classical HOV,taking advantage of tests already performed in previousinvestigations and considering that many ground pointswere available. Nevertheless, the procedure discussedhereafter may be obviously applied to LOOCV derivedaccuracy indices.

In order to represent the accuracy trend versus theGCPs number, a simple model based on a decreasingexponential function has been implemented:

y = a exp(bx−t) (3)

where y is the CPs RMSE, x is the number of GCPsused to build the model, a and b are estimated withtheir standard deviations (σa, σb) in a least squaresadjustment. The value of t is constrained to be positiveinteger and computed iteratively starting with t = 1 andestimating the two coefficients a and b; if the differencefrom the asymptotic value at x = nGCP/2 (that isy(nGCP/2) − a), is larger than a chosen threshold(e.g. >1 cm), the value of t is increased of a unit andthe estimates of a and b have to be recomputed. Thischoice to constrain the slope of the function to theasymptotic value has been done to avoid that possible

Fig. 1. Example of RMSE fit.

false oscillations of the estimated accuracy on fewCPs can affect the estimation of a and b parameters(Fig. 1). The asymptotic value (a ± σa ,) enables thedetermination of the number of GCPs (nGCP) sufficientto achieve the maximum accuracy, that is the integervalue larger than the x value corresponding to y =

a + 2σa (2-sigma upper confidence limit):

nGCP = int sup t

√b

ln(a + 2σa)− ln a. (4)

The overall GCPs number suitable to exploit themaximum accuracy is ranging between 10 and 20 (17GCPs were considered for both EROS-A images, 13GCPs for both QuickBird ones), depending on theimagery GSD, the off-nadir angle and its first derivative.Of course, this preliminary investigation could becarried out thanks to the large number of ground pointsavailable for all images, which is not generally the casein usual situations.

Next we applied the LOOCV to evaluate theaccuracy of each image in an alternative way by


using the same GCPs sets suitable to exploit themaximum accuracy according to the HOV. LOOCVaccuracies were evaluated through both the RMSEand the mAD and compared to the standard HOVRMSE; the comparison was not performed becauseHOV RMSE represents the best index (the goal of ourwork is just to show that it is likely to be a poor index, iffew ground points are available), but because this indexis the commonly adopted one.

Anyway it has to be stressed that LOOCV indicesand HOV RMSE may be really compared if computedconsidering CPs distributed on similar areas: for theLOOCV this area is just equal to the one covered byGCPs, whilst for HOV it depends on the distribution ofthe independent CPs. In fact, it has to be expected thatthe smaller is the area covered by GCPs, the better arigorous model (if estimable) behaves inside the areaitself.

The fundamental matter to deal with was to assesshow well LOOCV indices (mAD and RMSE) are ableto represent the overall accuracy; the accuracy indiceswere computed both for the North and East residualsseparately and for the horizontal residuals (modules,that is sqrt{(North residual) ˆ 2 + (East residual) ˆ 2});the maximum residual absolute values were alsoconsidered.

To test the proposed method we implemented a newroutine performing the LOOCV in the SISAR software;this routine was tested on some EROS-A and QuickBirdimages, as mentioned before. Moreover, these imageswere also oriented by the world recognized commercialsoftware OrthoEngine v. 10 (PCI Geomatica), manuallyperforming LOOCV since only HOV is possible withthis software. The software comparison guaranteedabout the overall correctness and good performances ofthe SISAR model, whereas the results showed the goodfeatures of the LOOCV method.

A second test was repeated for each image with adifferent set of GCPs in order to check the generality ofthe first results. Since the outcome is mainly the same,for the sake of brevity we decided to present only theformer.

In the discussion of the following results, mainlyabsolute horizontal residuals and related statistics areconsidered.

5.1. EROS-A images

Global results (Fig. 2 and Table 2, Fig. 3 and Table 3)showed that LOOCV mADs are generally lower thanHOV RMSEs, especially when large residuals inflateit. For instance, the analysis highlighted that the same

Fig. 2. Horizontal residual absolute values on CPs for ITA1-e1038452 image.

Fig. 3. Horizontal residual absolute values on CPs for ITA1-e1090724 image.

ground point (point 5) is critical for both softwarepackages, but in different images (Table 4); in thesecases the contribution of a robust accuracy index isremarkable, since LOOCV RMSEs are biased by theextremely high residuals on point 5 but in differentmanners. In any case, it has to be noted that when point5 acts as a GCP, no particular problems in the modelprecision and orientation accuracy is evidenced. Apossible explanation may be suggested by the positionof point 5, the northernmost along the perimeter of thearea covered by ground points (Fig. 4): the estimatedorientation models are significantly different if point 5acts (in HOV) or not (in LOOCV) as a GCP, so thatthey cannot be directly compared. This is why it seemsmore correct to filter out the effect of the anomalousresidual by LOOCV mAD instead of considering itas in LOOCV RMSE, since this point is not veryrepresentative of the mean achievable accuracy.

As concerns software comparison, since the imageITA1-e1090724 has a high off-nadir angle, softwarepackages behaved quite differently, SISAR being ableto achieve a remarkably better accuracy. Moreover, thenumber of LOOCV critical CPs, with North or East


Fig. 4. GCPs (dots) and HOV CPs (stars) for ITA1-e1038452 (left) and ITA1-e1090724 (right) images; LOOCV critical CPs with North or Eastresiduals larger than 3mAD are evidenced (triangles for SISAR, squares for OrthoEngine); areas covered by GCPs are delimited.

Table 2Comparison between models and accuracy indices (in pixels) for ITA1-e1038452 image

Accuracy index SISAR OrthoEngine

LOOCV North (pix) East (pix) Module (pix) North (pix) East (pix) Module (pix)RMSE 1.98 1.13 2.28 1.34 1.22 1.82mAD 1.03 0.79 1.41 0.50 0.91 1.35Abs max 6.24 2.05 6.29 3.30 3.06 3.79

HOV North (pix) East (pix) Module (pix) North (pix) East (pix) Module (pix)RMSE 1.50 1.30 1.98 1.07 1.17 1.58





residuals larger than 3mAD, is significantly lower forSISAR (1 in ITA1-e1038452 image on the perimeter)than for OrthoEngine (5 in ITA1-e1038452 image onthe perimeter, 2 in ITA1-e1090724 image both on theperimeter and inside the area covered by ground points).

5.2. QuickBird AUGUSTA image

No large residuals (Fig. 5) are evidenced in thiscase and LOOCV accuracies (RMSEs and mADs) areessentially of the same order of magnitude of HOVRMSE (Table 5), but again mADs are slightly lowerthan HOV RMSEs and they are significantly closer toHOV RMSEs than LOOCV RMSEs.

The number of LOOCV critical CPs (Fig. 6) is 1 forSISAR (on the perimeter) and 2 for OrthoEngine (1 on

Table 4Point 5 residual modules (in pixels)

Image SISAR (pix) OrthoEngine (pix)

ITA1-e1038452 6.29 3.50ITA1-e1090724 3.64 15.95

the perimeter and 1 inside the area covered by groundpoints).

5.3. QuickBird SALERNO image

In this case (Figs. 7 and 8) what has to be accountedfor is the length of the image and the not optimal spatialdistribution of the ground points, due to wide forests(unable to offer any element suitable to act as a groundpoint) in the northern part of the area. In this case, the


Table 5Comparison between models and accuracy indices (in pixels) for AUGUSTA image




Fig. 5. Residual absolute values on CPs for AUGUSTA image.

Fig. 6. GCPs (dots) and HOV CPs (stars); LOOCV CPs with Northor East residuals larger than 3mAD are evidenced (squares forOrthoEngine); area covered by GCPs is delimited.

availability of a “long” image is of particular value,so that also the image portion covering areas withoutGCPs (approximately as wide as a standard image) maybe oriented thanks to the GCPs distributed in otherparts. LOOCV mADs are similar for both packages andapproximately equal to HOV RMSEs, whilst LOOCVRMSEs are significantly higher (Table 6), probably dueto the irregular ground points distribution.

Fig. 7. Residual absolute values on CPs for SALERNO “tripleimage”.

5.4. Comparison LOO–(n − 1)GCPs HOV

According to what was explained in chapter 3,LOOCV results were obtained performing n iterationswith (n − 1) GCPs (each time the remaining groundpoint acts as a CP) whilst HOV results stemmed, asusual, from a unique test using all the available n GCPs.

In this respect LOOCV and HOV orientation modelsare different, LOOCV orientations being obtained withone GCP less than HOV. Therefore, these modelsmight be not completely comparable, especially whenexcluding one GCP from LOOCV model causes aremarkable decrease of the area covered by the GCPs.In these cases, LOOCV model with (n − 1) GCP mightsignificantly differ from the HOV model with n GCPs,and ground points acting as CPs quite external to thearea covered by GCPs might display high residuals,inflating the LOOCV RMSE but not mAD.

We compared the LOOCV mAD and RMSE tothe mean and median of the HOV RMSEs over then iterations, each with (n − 1) GCPs (Tables 7–10).The comparison shows that mAD is more suitablethan RMSE to represent the accuracy for the LOOCVmethod, since in all the tests the LOOCV mAD is muchcloser both to the HOV RMSE median with n and(n−1)GCPs than the LOOCV RMSE, generally higherbecause of the presence of anomalous residuals. It has


Table 6Comparison between models and accuracy indices (in pixels) for SALERNO “triple image”




to be underlined that the HOV RMSEs are computedconsidering only the CPs never used as GCPs (32 pointsfor EROS images, 26 and 42 points respectively for thetwo QuickBird images), not including in each iterationthe ground point acting as a CP in the LOOCV.

Nevertheless, the LOOCV mAD might appearalways better and too optimistic than the LOOCVRMSE. This is essentially due to the ability of theLOOCV mAD to filter out the higher residuals, mainlyattributable to the points placed along or outside thearea covered by GCPs; if we consider only the pointsacting as CPs inside this area (“internal points”) inorder to compute both LOOCV mAD and RMSE, weobtain much closer values for the two indices (Table 11).Therefore, if both indices are computed on a propersample (“internal points”), provided it is free fromoutliers, the results are in better agreement: a moreconservative indication could then be to use both theLOOCV mAD and the LOOCV RMSE to evaluate theaccuracy.

The RMSEs standard deviation over the n iterationsis also computed to evaluate the stability of theestimated models.

Results appear quite stable for SISAR, in the sensethat the n HOV RMSEs with (n − 1) GCPs are veryclose to each other (RMSE standard deviation are within11 cm) and also their mean/median are close to the HOVRMSE with n GCPs. In this case estimated models withn and (n − 1) seem quite close and the comparison withmAD leads to the same results as before.

As regards OrthoEngine, results appear sometimesignificantly different, as also evidenced by a muchhigher RMSE standard deviations, confirming a lowerstability.

Finally, in order to point out the intrinsic unreli-ability of the HOV RMSE when few ground pointsare available, we considered again the n orientationscarried out with (n − 1) GCPs but we computed theHOV RMSE on the basis of 4 CPs only. Two differ-ent CP sets were considered, located in different man-ners, named “internal” (EROS-A CPs: 18, 24, 31, 33;

Fig. 8. GCPs (dots) and HOV CPs (stars); LOOCV CP withNorth or East residuals larger than 3mAD is evidenced (square forOrthoEngine).

QuickBird AUGUSTA CPs: 1, 10, 14, 15; QuickBirdSALERNO CPs: 21, 35, 36, 46) and “external” (EROS-A CPs: 2, 8, 26, 52; QuickBird AUGUSTA CPs: 4, 5,7, 20; QuickBird SALERNO CPs: 3, 11, 15, 25) setrespectively (Figs. 2, 3, 5 and 7); here, only SISARresults are analyzed. In this way, it was easy to showthat HOV RMSE displays the risk to be too dependent




LOOCV North (pix) East (pix) Module (pix) North (pix) East (pix) Module (pix)RMSE 1.98 1.13 2.28 1.34 1.22 1.82mAD 1.03 0.79 1.41 0.50 0.91 1.35


HOV (n − 1) North (pix) East (pix) Module (pix) North (pix) East (pix) Module (pix)RMSE mean 1.52 1.32 2.02 1.08 1.17 1.59RMSE median 1.53 1.31 2.01 1.07 1.17 1.59RMSE std. dev. 0.07 0.03 0.05 0.06 0.06 0.06






Table 9Comparison between models and accuracy indices (in pixels) for AUGUSTA image





on the CPs distribution and may be biased when fewground points are available; in fact, the ground pointslocated along the perimeter are mostly used to esti-mate the orientation parameters, in order to achieve amore stable solution; therefore, CPs are likely to beconcentrated over a much narrower area than the onecovered by the image, so that the derived accuracy isnot representative for the whole image. Overall, RMSEmean appears underestimated if internal CPs are con-sidered (compared to those displayed in Tables 7–10),and vice versa happens for external CPs (Table 12), ex-cept from the EROS-A image with high off-nadir angle.

6. Conclusions

The paper deals with the application of a newmethod, the Leave-One-Out Cross-Validation (LOOCV),for the evaluation of HRSI orientation accuracy. Thismethod has already been applied in different fields suchas machine learning, bioinformatics and generally inany other field requiring an evaluation of the perfor-mance of a learning algorithm (e.g. in geostatistics),but never to HRSI orientation accuracy assessment. It isable to overcome the weaknesses of the currently usedHold-Out Validation (HOV), when the number of GCPs


Table 10Comparison between models and accuracy indices (in pixels) for SALERNO “triple image”





Table 11Comparison between global LOOCV mAD and LOOCV RMSE vs. LOOCV mAD and LOOCV RMSE computed on the “internal points”

Image Accuracy index Internal CPs All CPsNorth (pix) East (pix) Module (pix) North (pix) East (pix) Module (pix)

SISAR

EROS-A mAD 0.98 0.65 1.44 1.03 0.79 1.41ITA1-e1038452 RMSE 1.41 1.08 1.78 1.98 1.13 2.28EROS-A mAD 1.24 0.92 1.76 1.51 0.94 2.02ITA1-e1090724 RMSE 1.39 1.23 1.86 1.78 1.37 2.25QuickBird mAD 0.66 0.76 1.20 1.01 1.17 1.77AUGUSTA RMSE 0.98 0.88 1.32 1.51 1.45 2.09QuickBird mAD 1.32 0.61 1.48 1.45 0.61 1.57SALERNO RMSE 1.87 0.63 1.97 2.26 1.16 2.54

OrthoEngine

EROS-A mAD 0.50 0.87 1.05 0.50 0.91 1.35ITA1-e1038452 RMSE 1.07 0.88 1.39 1.34 1.22 1.82EROS-A mAD 1.49 1.23 2.50 2.32 1.26 3.22ITA1-e1090724 RMSE 2.46 1.47 2.86 4.60 2.03 5.03QuickBird mAD 0.99 0.42 1.14 1.19 0.96 1.68AUGUSTA RMSE 1.19 0.55 1.32 1.83 1.56 2.40QuickBird mAD 0.30 0.83 0.87 1.31 1.07 1.91SALERNO RMSE 0.98 0.85 1.31 2.37 1.29 2.70

Table 12HOV RMSE mean dependency on the CPs distribution (in pixels)

Image Internal CPs External CPs All CPs (RMSE mean)North (pix) East (pix) Module (pix) North (pix) East (pix) Module (pix) North (pix) East (pix) Module (pix)

EROS-A 1.76 1.29 2.18 1.06 2.55 2.78 1.52 1.32 2.02ITA1-e1038452EROS-A 2.73 1.50 3.12 2.58 1.75 3.13 2.11 1.69 2.71ITA1-e1090724QuickBird 0.66 0.98 1.19 1.55 1.59 2.21 1.27 1.38 1.88AUGUSTAQuickBird 1.04 0.55 1.19 1.75 1.16 2.10 1.23 0.78 1.50SALERNO

to be collected must be reduced as much as possible forlogistic and/or budgetary constraints; it could, therefore,

be a solution for this crucial problem in many practicalsituations.


As a matter of fact, in this case the HOV derivedaccuracy (RMSE of CPs residuals) is strictly driven bythe (very) few used CPs and may be not representativeof the overall mean accuracy, which is not reliableeither; moreover, it may be very sensitive to possibleoutliers affecting CPs coordinates, because it is notrobust. On the contrary, even in the presence of fewground points, LOOCV is able to capitalize all theavailable information all over the image.

The LOOCV method can be applied no matter whatthe orientation model is. In this work we just consideredand compared two different rigorous models, thoseimplemented in SISAR and OrthoEngine, due to thewider range of application of such a model type:the former has been developed by the Geodesy andGeomatics Team at the Sapienza University of Rome,the latter is a tool in a commercial well known package.Within SISAR an automatic procedure has beenimplemented to perform LOOCV, while OrthoEnginehas been manually used, since only the HOV procedureis included in it.

The fundamental matter to deal with was to assesshow well LOOCV derivable accuracy indices (mADand RMSE) are able to represent the overall accuracyand which are their advantages with respect to the HOVRMSE.

Some experiments were carried out over four images,two EROS-A Basic and two QuickBird Basic images,which led to the following main conclusions, pointingout that the LOOCV method with accuracy evaluatedby mAD and RMSE seems promising and useful forpractical cases:

• the LOOCV RMSE by itself and HOV RMSE aretoo sensitive to outliers and “critical” points (mainlylocated along the perimeter of the area covered byground points), which may display high residualswhen they act as CPs

• the HOV RMSE displays the risk to be too dependenton the geometric distribution of CPs, so that the HOVderived accuracy is likely to be not representative forthe whole image when few CPs are available

• the LOOCV mAD is a robust index able to filter outthe effect of the high residuals; this is of particularrelevancy for the “critical” points, which are notrepresentative of the mean achievable accuracy

• the LOOCV mAD and the LOOCV RMSE are ingood agreement if we consider only the points insidethe area covered by ground control points (“internalpoints”), provided they are free from outliers

• a simple decreasing exponential function wasproposed to represent the accuracy trend versus theGCPs number, and this model may be conveniently

applied to the LOOCV mAD to find the minimumnumber of GCPs for accuracy assessment when anumber of ground points is available.

As regards software comparison and SISAR perfor-mances assessment, the results are comparable, withsome differences in accuracy, especially in the case ofthe EROS-A image with high off-nadir angle; it is note-worthy that the differences are lower if the LOOCVmAD is considered instead of the HOV RMSE. Nev-ertheless, the comparison LOOCV–(n − 1) GCPs HOV(Section 5.4) showed that SISAR appears quite stable,in the sense that the n HOV RMSEs with (n − 1) GCPsdispersion (standard deviation) is very low and also theirmean/median are close to the HOV RMSE with n GCPs;on the contrary, in some cases OrthoEngine results ap-pear significantly different, which is also evidenced bya much higher RMSE standard deviations, testifying alower stability. In this respect, we can be confident aboutthe good performances of the SISAR model and the re-liability of the LOOCV derived results.

An additional problem evidenced by this investiga-tion is the evaluation of the accuracy uncertainty, inde-pendently from the used index, in order to enable a moreobjective comparison among the indices. This will beinvestigated in the future on the basis of Monte Carlosimulations, starting from the actual accuracy of groundpoints and images coordinates, estimating the desiredaccuracy index (e.g. mAD and RMSE) in several (thou-sands) trials and finally computing its uncertainty (stan-dard deviation).

Acknowledgements

The Authors thank very much the Editor, Prof. Eber-hard Gulch and the two Anonymous Reviewers, whosecomments and suggestion contributed to improve thepaper significantly.

This research was partially supported by grants ofthe Italian Ministry for School, University and ScientificResearch (MIUR) in the frame of the project MIUR-COFIN 2005 – “Analisi, comparazione e integrazionedi immagini digitali acquisite da piattaforma aerea esatellitare ” – National Principal Investigator: S. Dequal.

In the end, the Authors thank very much:Valerio Caroselli (Informatica per il Territorio S.r.l.,

Rome, Italy) and Fabio Volpe (Eurimage S.p.A., Rome,Italy), who kindly supplied the EROS-A and QuickBirdimagery respectively.

Laura De Vendictis, Francesca Lorenzon, LuciaLuzietti, Augusto Mazzoni, Roberta Onori, who carriedout the ground points GNSS surveys.


References

Amato, R., Dardanelli, V., Emmolo, D., Franco, V., Lo Brutto, M.,Midulla, P., Orlando, P., Villa, B., 2004. Digital orthophotosat a scale of 1:5000 from high resolution satellite images.International Archives of Photogrammetry, Remote Sensing andSpatial Information Sciences 35 (B4), 593–598.

Baiocchi, V., Crespi, M., De Vendictis, L., Giannone, F., 2004. Anew rigorous model for the orthorectification of synchronousand asynchronous high resolution imagery. In: Proceedings 24thEARSeL Symposium, Dubrovnik (Croatia), pp. 461–468.

Chen, L.C., Teo, T.A., 2002. Rigorous generation of digitalorthophotos from EROS-A high resolution satellite images.International Archives of Photogrammetry, Remote Sensing andSpatial Information Sciences 34 (4), 620–625.

Crespi, M., Giannone, F., Poli, D., 2006. Analysis of rigorousorientation models for pushbroom sensors. Applications withQuickBird. International Archives of Photogrammetry, RemoteSensing and Spatial Information Sciences 36 (1), on CD-ROM(reviewed section).

Di, K., Ma, R., Li, R., 2003. Rational functions and potential forrigorous sensor model recovery. Photogrammetric Engineeringand Remote Sensing 69 (1), 33–41.

Elisseeff, A., Pontil, M., 2002. Leave-one-out error and stability oflearn-ing algorithms with applications. In: Advances in LearningTheory: Methods, Models and Applications. NATO AdvancedStudy Institute on Learning Theory and Practice, pp. 111–130.

Fraser, C.S., 2003. Prospects for mapping from high-resolutionsatellite imagery. Asian Journal of Geoinformatics 4 (1), 3–10.

Fraser, C.S., Hanley, H.B., 2005. Bias compensated RPCs for sensororientation of high-resolution satellite imagery. PhotogrammetricEngineering and Remote Sensing 71 (8), 909–915.

Fraser, C.S., Dial, G., Grodecki, J., 2006. Sensor orientation via RPCs.ISPRS Journal of Photogrammetry and Remote Sensing 60 (3),182–194.

Geisser, S., 1975. The predictive sample reuse method withapplications. Journal of the American Statistical Association 70(350), 320–328.

Golub, G.H., Van Loan, C.F., 1993. Matrix Computation. The JohnsHopkins University Press, Baltimore and London.

Jacobsen, K., 2002. Comparison of high resolution mapping fromspace. Indian Cartographer 22, 106–119.

Kaula, W.M., 1966. Theory of Satellite Geodesy. Blaisdell PublishingCompany.

Kohavi, R., 1995. A study of cross-validation and bootstrap foraccuracy estimation and model selection. http://citeseer.ist.psu.edu/kohavi95study.html (accessed 21.09.07).

Noerdlinger, P.D., 1999. Atmospheric refraction effects in Earthremote sensing. ISPRS Journal of Photogrammetry & RemoteSensing 54 (5), 360–373.

Noguchi, M., Fraser, C.S., Nakamura, T., Shimono, T., Oki,S., 2004. Accuracy assessment of QuickBird Stereo Imagery.Photogrammetric Record 19 (106), 128–137.

Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P., 1992.Numerical Recipes in C: The Art of Scientific Computing, Cam-bridge University Press. http://www.nr.com (accessed 10.04.06).

Simon, R., Dobbin, K., McShane, L.M., 2003. Pitfalls in the use ofDNA microarray data for diagnostic and prognostic classification.JNCI Journal of the National Cancer Institute 2003 95 (1), 14–18.

Stone, M., 1974. Cross-validatory choice and assessment of statisticalpredictions (with discussion). Journal of the Royal StatisticalSociety B 36 (2), 111–147.

Strang, G., Borre, K., 1997. Linear Algebra, Geodesy and GPS.Wellesley-Cambridge Press, Wellesley.

Toutin, T., Cheng, P., 2002. QuickBird – A milestone for highresolution mapping. Earth Observation Magazine 11 (4), 14–18.

Toutin, T., Chenier, R., Carbonneau, Y., 2003. 3D models forhigh resolution images: Examples with QuickBird, IKONOSand EROS. In: Proc. ISPRS Commission IV Symposium, JointInternational Symposium on Geospatial Theory, Processing andApplications, Ottawa, 8–12 July, pp. 547–551.

Toutin, T., 2004. Geometric processing of remote sensing images:Models, algorithms and methods (review paper). InternationalJournal of Remote Sensing 25 (10), 1893–1924.

Westin, T., 1990. Precision rectification of SPOT imagery.Photogrammetric Engineering and Remote Sensing 56 (2),247–253.

Zhang, L., 2005. Automatic Digital Surface Model (DSM) generationfrom linear array images, Diss. ETH No. 16078, TechnischeWissenschaften ETH Zurich, 2005, IGP Mitteilung N. 90.

http://citeseer.ist.psu.edu/kohavi95study.html







http://www.nr.com

http://www.nr.com

http://www.nr.com

http://www.nr.com

Documents

Accuracy assessment of high resolution satellite imagery orientation by leave-one-out method