RegionalError EstimationofSurrogates(REES)c 2012 by Achille Messac. Published by the American Institute of Aeronautics and Astronautics, Inc. with permission. 1 of 14 American Institute

Regional Error Estimation of Surrogates (REES)

Ali Mehmani∗

Syracuse University, Syracuse, NY, 13244

Souma Chowdhury† and Jie Zhang †

Rensselaer Polytechnic Institute, Troy, New York 12180

Achille Messac‡

Syracuse University, Syracuse, NY, 13244

Surrogate-based design is an effective approach for modeling computationally expensivesystem behavior. In such application, it is often challenging to characterize the expectedaccuracy of the surrogate. In addition to global and local error measures, regional errormeasures can be used to understand and interpret the surrogate accuracy in the regions ofinterest. This paper develops the Regional Error Estimation of Surrogate (REES) methodto quantify the level of the error in any given subspace (or region) of the entire domain,when all the available training points have been invested to build the surrogate. In thisapproach, the accuracy of the surrogate in each subspace is estimated by modeling thevariations of the mean and the maximum error in that subspace with increasing number oftraining points (in an iterative process). A regression model is used for this purpose. Ateach iteration, the intermediate surrogate is constructed using a subset of the entire train-ing data, and tested over the remaining points. The evaluated errors at the intermediatetest points at each iteration are used for training the regression model that represents theerror variation with sample points. The effectiveness of the proposed method is illustratedusing standard test problems. To this end, the predicted regional errors of the surrogateconstructed using all the training points are compared with the regional errors estimatedover a large set of test points.

Keywords: Kriging; Regional error measures; Sampling; Surrogate models;

I. Introduction

A. Surrogate Modeling

Engineering design problems often involve computationally intensive simulation models (high fidelity models)or expensive experiment-based system evaluations. An accurate surrogate model is an effective tool forproviding a tractable and an inexpensive approximation of the actual system evaluation. During the lasttwo decades, the use of mathematical approximation models in design space exploration, sensitivity analysis,and optimization has become popular for reducing the computational cost and filtering the numerical noiseof computationally intensive system evaluations.1 This approximation model is known as a Surrogate orMetamodel (model of the models).2 Popular surrogate modeling methods include polynomial responsesurfaces,3 Kriging,4, 5 Moving Least Square,6, 7 radial basis functions,8 neural networks,9 and hybrid surrogatemodeling.10, 11 These methods have been applied to a wide range of disciplines, such as aerospace design,

∗Doctoral Student, Multidisciplinary Design and Optimization Laboratory, Department of Mechanical and Aerospace Engi-neering. AIAA Student Member

†Doctoral Student, Multidisciplinary Design and Optimization Laboratory, Department of Mechanical, Aerospace and Nu-clear Engineering. AIAA Student Member

‡Distinguished Professor and Department Chair. Department of Mechanical and Aerospace Engineering. AIAA LifetimeFellow. Corresponding author. Email: [email protected]

Copyright c© 2012 by Achille Messac. Published by the American Institute of Aeronautics and Astronautics, Inc. withpermission.

1 of 14

American Institute of Aeronautics and Astronautics

12th AIAA Aviation Technology, Integration, and Operations (ATIO) Conference and 14th AIAA/ISSM17 - 19 September 2012, Indianapolis, Indiana

AIAA 2012-5707

Copyright © 2012 by Achille Messac. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.

Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

automotive design, chemistry, and material science.12 The four main steps typically involved in constructing asurrogate model are: (i) choosing an appropriate method for performing the design of experiments (DoE); (ii)performing system evaluation using high fidelity simulation or physical experiment at the sample points; (iii)determining a surrogate model to fit the responses in the previous step; and (iv) evaluating the accuracy ofthe surrogate model. In the last step, error measures are used to assess the performance of the surrogate. Theerror measures can be also used for (i) domain exploration to identify interesting regions of a design space;(ii) adaptive sampling for surrogate development;13, 14 and (iii) surrogate-based design optimization.15, 16

Different error measures can be applied to assess the accuracy of surrogates. In the following subsection, thepopular error measurement methods for assessing the accuracy of surrogates are reviewed.

B. Review of Surrogate Model Error Measurement Methods

In this section, the methods for assessing errors in surrogates are reviewed. Error quantification methodscan be broadly classified, based on their computational expense, into (i) methods that require additionaldata, and (ii) methods that use existing data.17 Expensive methods are typically using additional data tocompare the surrogate’s response to the actual response. When extra data is not available, inexpensivemethods use existing data (the training data for constructing of the surrogate) to quantify the level of errorsin surrogates. Error measure methods can also be classified into global and local error estimations.18 Theoverall performance of the developed surrogates can be evaluated using global error measures, while the localor point-wise error measures evaluate the surrogate in different location of the surrogates.

Popular approaches of global error measures include:10 (i) split sample, (ii) cross-validation, and (iii)bootstrapping. It should be noted that these techniques are model independent. In split sample strategy,the sample data is divided into training and test data. The former is used to construct a surrogate; andthe latter is used to test the performance of the surrogate. The cross-validation is a popular technique toestimate the error of a developed surrogate. In Leave-one-out cross-validation, the training set is createdby taking all sample points except one, and the left out point is used for estimating error between thesurrogate prediction and the actual value. In q-fold cross-validation, the data set is split randomly into q(approximately) equal subsets. The surrogate is constructed q times, each time leaving out one of the subsetsfrom training points. The omitted subset, in each iteration, is used to evaluate the cross-validation error.19

The bootstrapping approach generates m subsamples from the sample points. Each subsample is a randomsample with replacement from the full sample. Different variants of the bootstrapping approach can be usedfor (i) model identification, and (ii) identifying confidence intervals for surrogates.10

Local approaches of error measures are used to estimate the level of errors in specific locations of a designspace. Examples of the local error measures include: (i) the mean square errors for Kriging15 and (ii) thelinear reference model (LRM).20 In stochastic surrogate models like Kriging, the errors at two different pointsof the design domain are not independent; and the correlation between the points is related to the distancebetween them. In particular, when the distance between the two points is small, the correlation is close toone; and when the distance is large, the correlation tends to zero. According to this correlation strategy, ifthe point x∗ is close to sample points, the prediction confidence at the point x∗ is much more than it wouldbe if x∗ was far away from all the sample points. This concept is reflected in the local error measurementmethod for Kriging predictor at the special point x∗. This error is equal to zero at sample points and isequal to

√σ2 at a point far away from sample points, where σ2 is the approximation error variance in the

stochastic process. The LRM is a model independent method for quantifying the local performance of asurrogate. The LRM considers the region with oscillations (complex behavior) as a high-error location. Thismethod categorizes errors of a surrogate in the design domain based on the deviation of the surrogate fromthe local linear interpolation.20

Besides the error measures mentioned above, there are other error metrics: (i) mean squared error (MSE)or root mean square error (RMSE) which provides a global error measure over the entire design domain,and (ii) maximum absolute error (MAE) and relative absolute error (RAE), which are indicative of localdeviation.

C. Research Objectives

The global measures, which can be helpful for representing the overall performance of the surrogates, maynot adequately represent the regional accuracy of a complex system. The primary objective of this paper isto develop an effective methodology for quantifying the surrogate error in each region of the design domain;

2 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

this method is called the Regional Error Estimation of Surrogate (REES). The proposed regional errormeasure method is model independent (it is not limited to a specific kind of surrogate model). The REESuses the available training data set to predict the error in different regions without any additional systemevaluations. This measure method provides useful information about the performance of the surrogate indifferent regions, thereby improving the useability of the surrogates in real life engineering design problems.

The paper is organized as follows: Section II describes the developed formulation of the proposed regionalerror measure method. In Section III, the REES is evaluated by two two-dimensional standard test problems.The results are discussed in Section IV. Concluding remarks and future work are provided in Section V.

II. Regional Error Estimation for Surrogate

A. Regional Error Estimation of Surrogate (REES) Measure

The goal of the proposed approach is to provide a regional measure of the level of errors in an estimatedfunction without investing additional (expensive) system evaluations. The REES method estimates theerror of the surrogate in a given region by predicting the variation of the error in that region with increasingnumber of training points, which the total number of available training points remains fixed. The size ofthe training data set is increased in each step while the remaining sample points are used as test points toevaluate the error. The framework of the proposed methodology is illustrated in Fig. 1, and is describedbelow:

Step 1. Generation of Training Data In this step, a set of experimental designs are chosen to cover theentire design space as uniformly as possible. Then, the system is evaluated over selected test datapoints. The entire set of sample points is represented by {X}.

Step 2. Estimation of the Variation of the Error in Different Regions This step consists of an it-erative process. In each iteration, the sample points are divided into a tentative training data set{Xt

TR} and a tentative test data set {XtTE}, given by

{XtTR} = {Xt−1

TR }+ {XtTRA} (1)

where (2)

{X0TR} = {}, {X1

TRA} ⊂ {X}, and {XtTRA}t>1 ⊂ {Xt−1

TE }The tentative set of training points {Xt

TR} at the iteration t consists of the previous set of trainingpoints {Xt−1

TR } and the current additional selected points {XtTRA}. The additional selected points are

a subset of the full set of the training data in the first iteration, and they are a subset of the test pointsin subsequent iterations.

The intermediate surrogate (ft) at each iteration is constructed using the tentative training pointsand is evaluated over the tentative test points. Then, domain segmentation strategy is applied todivide the design space into subspaces. Based on domain segmentation, the whole design domain isdivided into physically meaningful classes. The boundary between classes can be determined usingerror-based pattern classification methods21 or can be predefined by the designer based on the level ofthe knowledge regarding the design problem. The mean error (Et

M ) and the maximum error (EtMax)

are then estimated at each iteration, as given by

EtM = [EMean

i ], EMeani = Mean(ei1, ei2, ..., eimi

),

EtMax = [EMax

i ], EMaxi = Max(ei1, ei2, ..., eimi

), (3)

where i = 1, ..., n

In Eq.3, the eij is the error of the jth test point in the ith subspace at iteration t. The parameter nrepresents the number of subspaces and mi is the number of test points in the ith subspace at the tth

iteration.

Step 3. Final Surrogate Construction and Error Prediction in Different Regions The final sur-rogate model is constructed using the full set of training data. A regression model is developed topredict the level of errors in each region of the final surrogate. This regression model, called the Vari-ation of the Error with Sample Points (VESP), is trained using the mean and the maximum errorinformation from all the preceding iterations.

3 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

Generate a set of Sample Points

Full set of Training Data,

Sub-set of Training Data,

1{ } { } { }t t t

TR TR TRAX X X-= +

where 0{ } {}, andTRX =

Checking if { } { }tTRX Xº

Construction of Intermediate Surrogate, f t

Estimation of error of f t at test points, 3{ }t

TEX

Estimation of

Mean ( ) and Max Error ( )

in each of the subspaces at iteration t

tME t

MaxE

Pool of test points

Segregation of

domain into

subspaces

{ } { } { }t tTE TRX X X= -

Construction of Final

Surrogate

Training VESP Regression

Model using and 3

Prediction of Error (Max

and Mean) in different

subspaces of the final

surrogate

ME MaxE

System Evaluation(High fidelity simulation or experiment)

Yes

No

t=t+1

Step 3Step 2

Step 1

Input Data

Output Data

1{ } { }t tTETRAX X -

Ì

{ }X

Figure 1. The framework of the Regional Error Estimation of Surrogate (REES)

B. Implementation of the REES Measure

For the sake of simplicity, a two-dimensional problem is used to illustrate the implementation of the proposedmethod. The total number of sample points (nX) allowed for this function is 32. The first tentative surrogate(at t = 1) is constructed using 8 training points, out of the 32 points; and each subsequent surrogate isconstructed using additional 8 points ({Xt

TRA}). The sampling scheme developed by Audze-Eglais22 isadopted to determine the locations of the full set of sample points ({X}). It is a Latin Hypercube-baseddesign of experiments method that is accompanied by an optimality criterion as given by

P∑

p

P∑

q=p+1

1

L2pq

→ min (4)

where Lpq is the distance between the points p and q, and P is the total number of sample points. TheLatin Hypercube based on max-min criterion is used for selecting additional points ({XTRA}) from the totalsample set ({X}) in each subsequent iteration. This criterion seeks to maximize the minimum distancebetween selected points. In each iteration, the set of training points consist of selected additional points{Xt

TRA} and the training points from the previous iteration {Xt−1TR }. The training points and the test points

for the first, second, third, and forth iteration (before termination of iterative process at nXt

TR= 32) are

illustrated in Figs. 2(a), 2(b), 2(c), and 2(d), respectively.The surrogates in this paper are developed using Kriging method, which is a widely used surrogate

modeling method. The standard Kriging model is given by

f(x) = G(x) + Z(x) (5)

where f(x) is the approximation of the actual response f(x); G(x) is the known type of approximationfunction (often a polynomial); and Z(x) is a weak stationary stochastic process with zero mean and varianceσ2.23 The ijth element of the covariance matrix of Z(x) is given by

COV [Z(xi), Z(xj)] = σ2zRij (6)

4 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

x1

x 2

(a) First iteration (nXTR=8, nXTE

=24)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

x1

x 2

(b) Second iteration (nXTR=16, nXTE

=16)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

x1

x 2

(c) Third iteration (nXTR=24, nXTE

=8)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

x1

x 2

(d) Fourth iteration (nXTR=32)

Figure 2. Tentative Training and Test Points in 2D Function (the red data points represent training pointsand blue data points represent test points)

5 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

where Rij is the correlation function between the ith and the jth data points; and σ2z is the process variance.

In this paper, a Gaussian function is used as the correlation function. The intermediate surrogate model ateach iteration is tested on the set of test points at that iteration. The Relative Accuracy Error (RAE) isused to evaluate the error at each test point. The RAE for test point xi is defined as

RAE(xi) =

∣

∣

∣

∣

∣

f(xi)− f(xi)

f(xi)

∣

∣

∣

∣

∣

(7)

where f(xi) is the actual function value at xi, and f(xi) represents the function value estimation by thesurrogate model. The mean and maximum values of the error in each subspace of the domain are determined

Figure 3. Illustration of dividing domain design into four equal rectangular subspaces in the two-dimensionalfunction

from the RAE estimated at all test points allowed at the concerned iteration. In the current paper, the entiredesign space is manually divided into a specific number of subspaces. In the case of the two-dimensionalfunction, the design space is divided into four equal rectangular subspaces as shown in Fig. 3. The errors(EMean and EMax) in each subspace estimated during the iterative process are used to train the VESPregression model. The selection of the type of regression function is critical to the prediction of the regionalerror levels in the surrogate. In this paper, we explore an exponential regression function to train the VESPmodel, which is expressed as

F (x, a) = a0e(a1 n

XtTR

)(8)

where a0 and a1 are unknown coefficients, and nXt

TRrepresents the number of training points in the tth

iteration. In the next section, the proposed method is applied to two benchmark test problems.

III. Numerical Examples and Testing Procedure

The performance of the Regional Error Estimation of Surrogate (REES) method is evaluated using thefollowing analytical test problems: (i) Dixon & Price function, and (ii) Branin function.

Test Function 1 : Dixon & Price Function

f(x) = (x1 − 1)2+ 2

(

2x22 − x1

)2(9)

where x1 ∈ [−10 10], x2 ∈ [−10 10]

Test Function 2 : Branin Function

f(x) =

(

x2 −5

4π2x21 +

5

πx1 − 6

)2

+ 10

(

1− 1

8π

)

cos(x1) + 10 (10)

where x1 ∈ [−5 10], x2 ∈ [0 15]

6 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

Numerical settings of two problems are given in Table 1. The table lists (i) the number of input variables,(ii) the total number of sample points nX , and (iii) the number of training (nXt

TR) and test points (nXt

TE) at

each iteration. For the two test problems, the first tentative surrogate model is constructed using 16 points

Table 1. Numerical setup for test problems

FunctionNo. of Total No. of

Iteration (t)No. of training points No. of test points

variables sample points (nX) at each iter. (nXt

TR) at each iter. (nXt

TE)

Dixon & Price 2 32

1 16 16

2 20 12

3 24 8

4 28 4

Branin 2 32

1 16 16

2 20 12

3 24 8

4 28 4

(nX1

TR= nX1

TRA). The number of additional points at each iteration is defined to be 4 (nXt

TRA= 4, t > 1).

The remaining sample points at each iteration ({X} − {XtTR}) are used as test points to evaluate the

performance of the tentative surrogate in each predefined subspace. For the two problems, the designspace is divided into four equal rectangular subspaces (as illustrated in Fig. 3). The Kriging surrogatemodel is developed for each problem and the surrogate accuracy is evaluated using the REES method. Toimplement the Kriging method, the DACE (design and analysis of computer experiments) package developedby Lophaven et al.24is used. The bounds on the correlation parameters in the nonlinear optimization, θl andθu, are specified to be 0.1 and 20, respectively. The order of the global polynomial trend function is specifiedto be zero.

IV. Results and Discussion

Before implementing the Regional Error Estimation of Surrogate (REES) method on the test problems,the actual performance of the final surrogate in each subspace is evaluated using Relative Accuracy Error(RAE) between surrogate predictions and actual values on larger set of test points. To this end, differentsets of test points ranging from 100 to 15,000 points are studied. The optimal Latin Hypercube based onthe max-min criterion25 is adopted to determine the locations of test points for each set. The performanceof the final surrogate on 100, 500, 1000, and 10,000 test points are illustrated in Figs. 4-7 for the Dixon &Price function, and in Figs. 8-11 for the Branin function. In this paper, RAEs are classified into four levelsbased on the estimated error. The predefined lower and upper bounds of each level for the mean and themaximum errors are defined as

EMeani ∈

Level 1 if EMeani ≤ 0.5µMean

Level 2 if 0.5µMean < EMeani ≤ µMean

Level 3 if µMean < EMeani ≤ 1.5µMean

Level 4 if EMeani > 1.5µMean

(11)

where µMean = Mean(EMeani ), and i=1,...,n

EMaxi ∈

Level 1 if EMaxi ≤ 0.5µMax

Level 2 if 0.5µMax < EMaxi ≤ µMax

Level 3 if µMax < EMaxi ≤ 1.5µMax

Level 4 if EMaxi > 1.5µMax

(12)

where µMax = Mean(EMaxi ), and i=1,...,n

7 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

In Eqs. 11 and 12, EMeani and EMax

i are normalized values of the mean and maximum errors evaluatedin the ith subspace, respectively. Interesting observations are made from Figs. 4-11: (i) The error levelclassifications changes when the number of test points is increased from 100 to 10,000. We observe thatsurrogate performance evaluated on a specific number of test points (i.e., 100 or 200 test points) does notnecessarily represent the actual performance of the surrogate. In these two test problems, the error levelclassification does not change extremely beyond 1000 test points. It should be noted that this number mayvary depending on complexity of the problems. (ii) According to the difference among the level of errorsin each subspace of the design domain, global error measures might not provide adequate knowledge of thesurrogate reliability in the process of the surrogate-based design.

(a) Mean of the RAE in each subspace - EMean (b) Maximum of the RAE in each subspace - EMax

Figure 4. The error levels of the surrogate in different subspaces on 100 Test Points (Dixon & Price function)



In REES method, the VESP regression model in each subspace is used to predict the level of the errorin that subspace of the final surrogate. The trends of the VESP models for the Dixon & Price and theBranin functions are illustrated in Figs. 12 and 14, respectively. The VESP regression model is trained bythe mean and the maximum errors evaluated for the intermediate surrogates. These intermediate surrogatesare constructed and tested using the tentative training and test points defined in Table 1. Expectedly, thelevel of the error decrease with increasing the number of training points. It is observed that the RAE incertain iterations is relatively higher than that in preceding iteration (i.e., second iteration in 12(b)). Thecoefficients of the exponential regression function (Eq. 8) used in the VESP model for the two test problemsare given in Tables 2 and 3.

The regional errors (maximum and mean) predicted based on the REES method in each subspace are

8 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07






Figure 8. The error levels of the surrogate in different subspaces on 100 Test Points (Branin function)

9 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07







10 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

Table 2. VESP coefficients in each subspace for Dixon & Price function

RegionVESP for EMean VESP for EMax

a0 a1 a0 a1

Subspace 1 2203.4515 -0.3050 5991.1503 -0.3087

Subspace 2 7.4808 -0.0041 17.9141 -0.0116

Subspace 3 126.8177 -0.1999 215.1334 -0.1960

Subspace 4 209.6296 -0.1048 516.8065 -0.0978

Table 3. VESP coefficients in each subspace for Branin function

RegionVESP for EMean VESP for EMax

a0 a1 a0 a1

Subspace 1 2.3351 -0.0634 10.00 -0.0923

Subspace 2 0.2650 -0.0413 0.6625 -0.0627

Subspace 3 0.7180 0.0021 1.5777 -0.0276

Subspace 4 3.9213 -0.0057 16.6621 -0.0215

illustrated in Figs. 13 and 15. The predicted RAEs are classified into four levels defined in Eqs. 11 and 12.For the Dixon & Price function, the combination of the predicted performance (Fig. 13) with the actualperformance evaluated by 10000 test points (Fig. 7) shows that the REES method successfully predictsthe level of the error (mean and max) in each subspace. In Branin function, the comparison betweenthe predicted performance and the actual performance evaluated by 10000 test points shows that: (i) thepredicted maximum error levels (Fig. 15(b)) are identical to the actual maximum error levels (Fig. 11(b));and (ii) there is slightly difference between the predicted and the actual levels of mean error (Fig. 15(a) andFig. 11(a)). Overall, the REES method is capable of locating the region with the high level of error.

V. Conclusion

This paper developed a new methodology to quantify the surrogate error in the different regions ofthe design domain, which is called the Regional Error Estimation of Surrogate (REES) method. The REESmethod provides a model independent error measure that does not require any additional system evaluations.In the REES method, after segregating the design space into subspaces (or regions), Variation of the Errorwith Sample Points (VESP) regression models are constructed to predict the accuracy of the surrogate in eachsubspace. These regression models are trained by the errors (the mean and the maximum error) evaluatedfor the intermediate surrogates in an iterative process. At each iteration, the intermediate surrogate isconstructed using different subsets of training points and tested over the remaining points. In this paper,the boundaries of the subspaces are predefined manually, and an exponential regression function is used totrain the VESP model. Two numerical test problems are examined to evaluate the predicted performanceof the REES measure. The preliminary results indicate that the REES measure is capable of evaluating theregional performance of a surrogate with reasonable accuracy.

Future work may include applying pattern classification methods to segregate the domain space at eachiteration based on the evaluated error, instead of pre-specifying subspace boundaries.

VI. Acknowledgements

Support from the National Science Foundation Awards CMMI 0946765 and CMMI 1100948 is gratefullyacknowledged. Any opinions, findings, and conclusions or recommendations expressed in this paper are those

11 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

15 20 25 30 350

10

20

30

40

50

Number of Sample Points (N)

RA

E

MeanMax

EMean

= 2203.45 e−0.3N

EMax

= 5991.15 e−0.31N

(a) Subspace 1

15 20 25 30 350

5

10

15

20

25

30

35


RA

E

MeanMax

EMean

= 7.48 e−0.004N

EMax

= 17.92 e−0.01N

(b) Subspace 2

15 20 25 30 350

2

4

6

8

10


RA

E

MeanMax

EMax

= 215.13 e−0.2N

EMean

= 126.82 e−0.2N

(c) Subspace 3

15 20 25 30 350

50

100

150


RA

E

MeanMax

EMax

= 516.81 e−0.1N

EMean

= 209.63 e−0.1N

(d) Subspace 4

Figure 12. The trend of VESP regression model in different subspaces (Dixon & Price function)


Figure 13. The predicted performance of the surrogate in different subspaces based on the REEM measure(Dixon & Price function)

12 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

15 20 25 30 350

0.5

1

1.5

2

2.5

3

3.5


RA

E

MeanMax

EMax

= 10 e−0.09N

EMean

= 2.33 e−0.06N

(a) Subspace 1

15 20 25 30 350

0.05

0.1

0.15

0.2

0.25

0.3

0.35


RA

E

MeanMax

EMax

= 0.66 e−0.06N

EMean

= 0.26 e−0.04N

(b) Subspace 2

15 20 25 30 350.4

0.6

0.8

1

1.2

1.4


RA

E

MeanMax

EMax

= 1.58 e−0.03N

EMean

= 0.72 e0.002N

(c) Subspace 3

15 20 25 30 352

4

6

8

10

12

14


RA

E

MeanMaxE

Max= 16.66 e

−0.02N

EMean

= 3.92 e−0.006N

(d) Subspace 4

Figure 14. The trend of VESP regression model in different subspaces (Branin function)


Figure 15. The predicted performance of the surrogate in different subspaces based on the REEM measure(Branin function)

13 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

of the authors and do not necessarily reflect the views of the NSF.

References

1Zadeh, P. M. and Mehmani, A., “Multidisciplinary Design Optimization Using Variable Fidelity Modeling: Applicationto a Wing Based on High Fidelity Models,” Third International Conference on Multidisciplinary Design Optimziation, Paris,France, June 2010.

2Kleijnen, J., Statistical techniques in simulation, Vol. II, New York, 1975, Russian translation, Publishing House Statistics,Moscow, 1978.

3Jin, R., Chen, W., and Simpson, T. W., “Comparative Studies of Metamodeling Techniques Under Multiple ModelingCriteria,” AIAA, , No. 4801, 2000.

4Simpson, T., Korte, J., Mauery, T., and Mistree, F., “Kriging Models for Global Approximation in Simulation-BasedMultidisciplinary Design Optimization,” AIAA Journal , Vol. 39, No. 12, 2001, pp. 2233–2241.

5Forrester, A. and Keane, A., “Recent Advances in Surrogate-based Optimization,” Progress in Aerospace Sciences,Vol. 45, No. 1-3, 2009, pp. 50–79.

6Choi, K., Young, B., and Yang, R., “Moving Least Square Method for Reliability-based Design Optimization,” 4th WorldCongress of Structural and Multidisciplinary Optimization, Dalian, China, June 2001, pp. 4–8.

7Toropov, V. V., Schramm, U., Sahai, A., Jones, R. D., and Zeguer, T., “Design Optimization and Stochastic Analysisbased on the Moving Least Squares Method,” 6th World Congresses of Structural and Multidisciplinary Optimization, Rio deJaneiro, 30 May - 03 June 2005.

8Hardy, R. L., “Multiquadric Equations of Topography and Other Irregular Surfaces,” Journal of Geophysical Research,Vol. 76, 1971, pp. 1905–1915.

9Yegnanarayana, B., Artificial Neural Networks, PHI Learning Pvt. Ltd., 2004.10Queipo, N., Haftka, R., Shyy, W., Goel, T., Vaidyanathan, R., and Tucker, P., “Surrogate-based Analysis and Optimiza-

tion,” Progress in Aerospace Sciences, Vol. 41, No. 1, 2005, pp. 1–28.11Zhang, J., Chowdhury, S., and Messac, A., “An Adaptive Hybrid Surrogate Model,” Structural and Multidisciplinary

Optimization, Vol. 46, No. 2, 2012, pp. 223–238.12Wang, G. and Shan, S., “Review of Metamodeling Techniques in Support of Engineering Design Optimization,” Journal

of Mechanical Design, Vol. 129, No. 4, 2007, pp. 370–381.13Lehmensiek, R., Meyer, P., and Muller, M., “Adaptive sampling applied to multivariate, multiple output rational in-

terpolation models with application to microwave circuits,” International Journal of RF and Microwave Computer-AidedEngineering , Vol. 12, No. 4, 2002, pp. 332–340.

14Sugiyama, M., “Active Learning in Approximately Linear Regression Based on Conditional Expectation of GeneralizationError,” Journal of Machine Learning Research, Vol. 7, 2006, pp. 141–166.

15Jones, D., Schonlau, M., and Welch, W., “Efficient Global Optimization of Expensive Black-box Functions,” Journal ofGlobal Optimization, Vol. 13, No. 4, 1998, pp. 455–492.

16Mehmani, A., Zhang, J., Chowdhury, S., and Messac, A., “Surrogate-based Design Optimization with Adaptive SequentialSampling,” 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference, Hawaii, USA,April 2012.

17Meckesheimer, M., Barton, R., Simpson, T., and Booker, A., “Computationally inexpensive Metamodel AssessmentStrategies,” AIAA, Vol. 40, No. 10, 2002, pp. 2053–2060.

18Goel, T., Haftka, R., and Shyy, W., “Ensemble of Surrogates,” Structural and Multidisciplinary Optimization, Vol. 38,2009, pp. 429442.

19Viana, F., Haftka, R., and Steffen, V., “Multiple Surrogates: How Cross-validation Errors Can Help Us to Obtain theBest Predictor,” Structural and Multidisciplinary Optimization, Vol. 39, No. 4, 2009, pp. 439–457.

20Nguyen, H. M., Couckuyt, I., Knockaert, L., Dhaene, T., Gorissen, D., and Saeys, Y., “An Alternative Approach toAvoid Overfitting for Surrogate Models,” Proceedings of the 2011 Winter Simulation Conference, 2011, pp. 2765–2776.

21Zhang, J., Chowdhury, S., and Messac, A., “Domain Segmentation based on Uncertainty in the Surrogate (DSUS),” 53rdAIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference, Hawaii, USA, April 2012.

22Audze, P. and Eglais, V., “New Approach for Planning Out of Experiments,” Problems of Dynamics and Strengths,Vol. 35, 1997, pp. 104–107, in Russian.

23Cressie, N., Statistics for Spatial Data, Wiley, New York, 1993.24Lophaven, S. N., Nielsen, H. B., and Sondergaard, J., “DACE - A Matlab Kriging Toolbox, Version 2.0,” Tech. Rep.

IMM-REP-2002-12, Informatics and Mathematical Modelling Report, Technical University of Denmark, 2002.25Viana, F. A. C., Venter, G., and Balabanov, V., “An algorithm for fast optimal Latin hypercube design of experiments,”

International Journal for Numerical Methods in Engineering , Vol. 82, No. 2, April 2010, pp. 135–156.

14 of 14


Dow

nloa

ded

by A

chill

e M

essa

c on

Sep

tem

ber

20, 2

012

| http

://ar

c.ai

aa.o

rg |

DO

I: 1

0.25

14/6

.201

2-57

07

Documents

RegionalError EstimationofSurrogates(REES)c 2012 by Achille Messac. Published by the American Institute of Aeronautics and Astronautics, Inc. with permission. 1 of 14 American Institute