Response surface approximation of Pareto optimal front in multi-objective optimization

www.elsevier.com/locate/cma

Comput. Methods Appl. Mech. Engrg. 196 (2007) 879–893

Response surface approximation of Pareto optimal frontin multi-objective optimization

Tushar Goel a,*, Rajkumar Vaidyanathan a,1, Raphael T. Haftka a, Wei Shyy a,2,Nestor V. Queipo b, Kevin Tucker c

a Department of Mechanical and Aerospace Engineering, University of Florida, P.O. Box 116250, Gainesville, FL 32611-6250, United Statesb Applied Computing Institute, Faculty of Engineering, University of Zulia, Venezuela

c NASA Marshall Space Center, MS/TD64, MSFC, AL 35812, United States

Received 31 March 2005; received in revised form 6 February 2006; accepted 6 July 2006

Abstract

A systematic approach is presented to approximate the Pareto optimal front (POF) by a response surface approximation. The data forthe POF is obtained by multi-objective evolutionary algorithm. Improvements to address drift in the POF are also presented. Theapproximated POF can help visualize and quantify trade-offs among objectives to select compromise designs. The bounds of this approx-imate POF are obtained using multiple convex-hulls. The proposed approach is applied to study trade-offs among objectives of a rocketinjector design problem where performance and life objectives compete. The POF is approximated using a quintic polynomial. The com-promise region quantifies trade-offs among objectives.� 2006 Elsevier B.V. All rights reserved.

Keywords: Pareto optimal front; Response surface approximation; Multi-objective evolutionary algorithms; Rocket injector design; Pareto drift

1. Introduction

Practical engineering design often involves multiple dis-ciplines and lacks the benefit of closed form analytical solu-tions. The design scope is frequently defined by multiple andsometimes conflicting design objectives, along with a sub-stantial number of design variables. Multi-objective optimi-zation problems usually have many optimal solutions,known as Pareto optimal solutions [1]. Each Pareto optimalsolution represents a different compromise among designobjectives. Hence, the designer is interested in finding manyPareto optimal solutions in order to select a design compro-mise that suits his preference structure. There are a numberof different methods available for solving multi-objective

0045-7825/$ - see front matter � 2006 Elsevier B.V. All rights reserved.

doi:10.1016/j.cma.2006.07.010

* Corresponding author. Tel.: +1 352 392 6780; fax: +1 352 392 7303.E-mail address: [email protected] (T. Goel).

1 Currently with General Motors, Bangalore, India.2 Present address: Department of Aerospace Engineering, The Univer-

sity of Michigan, Ann Arbor, MI 48109, United Sates.

optimization problems. One popular approach is condens-ing multiple objectives into a single, composite objectivefunction by methods like weighted sum, geometric mean,perturbation, Tchybeshev, min–max, goal programming,and physical programming [1–3]. Another approach is tooptimize one objective while treating other objectives asconstraints [4]. These approaches give one Pareto optimalsolution in each simulation.

There are numerous multi-objective evolutionary algo-rithms (MOEAs) [5] that can be made to find multiple Par-eto optimal solutions in a single simulation run [5]. Some ofthe latest ones include strength Pareto evolutionary algo-rithm (Zitzler and Thiele) [6], Pareto archived evolutionarystrategies (PAES by Knowles and Corne) [7], elitist non-dominated sorting genetic algorithm (NSGA-II by Debet al.) [8], and controlled elitist non-dominated sortinggenetic algorithm (Deb and Goel) [9].

Most of the popular evolutionary algorithms are basedon the concept of Pareto dominance and involve a finitesize of population at each generation [5]. Due to the finite

mailto:[email protected]

880 T. Goel et al. / Comput. Methods Appl. Mech. Engrg. 196 (2007) 879–893

size of the population, some of the good solutions makeway for the other solutions. If the lost solutions are thePareto optimal solutions, the loss may not be repairedand sub-Pareto optimal solutions are obtained as the finalsolution. This problem is named here Pareto drift. A rem-edy for this problem is suggested in the form of maintain-ing an archive of the Pareto optimal solutions.

To solve multi-objective optimization problems one cancouple EAs with exact function evaluations or with surro-gate models of computationally expensive function evalua-tors. With reference to the former scenario, Sasaki et al.[10,11] used EAs coupled with CFD analysis to designthe supersonic wings for multiple objectives. Makinenet al. [12] used EAs coupled with CFD analysis to solve air-craft wing design problems. Deb and Goel [13–15] usedevolutionary algorithms and a posterior local search toobtain Pareto optimal solution set of simple mechanicalcomponent shapes. Ishibuchi et al. [16] used a combinationof evolutionary algorithms and local search for finding Par-eto optimal solutions in flowshop scheduling test problems.Obayashi et al. [17] used EAs coupled with CFD to designsupersonic wings. In all these works, evolutionary algo-rithms are directly coupled with exact evaluation ofdesigns. For computationally expensive problems, directcoupling of function evaluators with EAs will be impracti-cal because multi-objective evolutionary algorithms requiremany analyses. Therefore, surrogates such as responsesurface approximations are often adopted [18–35].

Once such surrogate models are available, the computa-tional burden of performing optimization and generatinga multitude of trade-off solutions is substantially reduced.For example, Madsen et al. [18], Vaidyanathan et al.[19,20], Shyy et al. [21] and Papila et al. [22,23] used poly-nomial- or neural network-based surrogate models asdesign evaluators for the optimization of propulsion com-ponents such as turbulent flow diffuser, supersonic turbineand swirl coaxial injector element. Dornberger et al. [24]used neural networks and polynomial response surfacesto approximate the design objectives and a modificationof genetic algorithm to find the Pareto optimal solutionsfor the design of turbine blades. Bramanti et al. [25] usedneural network models to approximate the design objec-tives and then coupled these models with evolutionaryalgorithm to find multiple trade-off solutions to electro-magnetic problems. Wilson et al. [26] and Cappelleriet al. [27] used surrogate modeling (response surfaceapproximations and kriging) for approximating the objec-tives while designing piezomorph actuators. Farina et al.[28,29] used evolutionary strategies along with multiquad-rics interpolation-based response surface approximationsto optimize the shape of electromagnetic components likeC-core and magnetizer for single and multiple objectives.Emmerich et al. [30,31] proposed using local metamodel

(Kriging approximation based on a few nearest neighbors)to evaluate objectives required for optimization of analyti-cal test functions and an airfoil shape design problem usingevolutionary algorithms. Ong et al. [32] used radial basis

functions to approximate the objective function and con-straints and then used a combination of evolutionary algo-rithm and sequential quadratic programming to findoptimal solutions of an aircraft wing design problem withsingle objective. Recently, Nain and Deb [33] combinedartificial neural networks and evolutionary algorithms toreduce the computational cost while finding trade-off solu-tions for standard multi-objective optimization test prob-lems. In a latest effort, Knowles and Hughes [34], andKnowles [35] combined a Gaussian process based globaloptimizer EGO with evolutionary multi-objective optimi-zation algorithms to significantly reduce the computationalcost in optimization of analytical test problems.

As previously stated, a major advantage of using surro-gate models is that function evaluation becomes inexpen-sive, making it feasible to evaluate a large number ofPareto optimal designs (e.g. [26]). Once many Pareto opti-mal solutions are obtained, the Pareto optimal front (POF)can be represented by its own response surface and thedomain of application can be identified. Though thedesigner has many Pareto optimal solutions, he still facesthe problem of selecting the compromise solution. Theresponse surface approximation of the Pareto optimalfront would allow the designer to visualize and assesstrade-offs among the objectives, to explore compromisesolutions, and to take decisions based on realistic goals.

This paper presents a methodology to construct aresponse surface approximation of the Pareto optimalfront based on surrogate models. The methodology is dem-onstrated using a four-objective single element gas–gasinjector design problem motivated by liquid-rocket injectorapplications. Major interests here are to probe the interac-tions and trade-offs among objectives, assessing correla-tions and conflicts among them. The objective functionevaluations require the solution of a fluid dynamics prob-lem involving turbulence and combustion. The solution isobtained by numerically solving the Navier–Stokes equa-tions, aided by the turbulence closure and chemical kineticschemes. A pressure-based, finite difference, Navier–Stokessolver, FDNS500-CVS [36–38], is used in this study. Steadystate is considered for all problems.

The objectives of this paper can be summarized asfollows:

1. demonstrate the construction of the Pareto optimalfront response surface approximation (Pareto optimalfront RSA), addressing the problem of its region ofapplicability in function space;

2. discuss the Pareto-drift problem and suggest a remedy inthe context of dominance based evolutionary algorithmsand

3. illustrate the objective function trade-offs for liquid-rocket injector design problem.

The paper is organized as follows: Section 2 gives thegeneral description of terms used in the paper. Section 3briefly describes the NSGA-II algorithm. Section 4

T. Goel et al. / Comput. Methods Appl. Mech. Engrg. 196 (2007) 879–893 881

discusses the problem of Pareto drift and proposes one pos-sible remedy. A method for constructing a response surfaceapproximation for the Pareto optimal front is outlined inSection 5 and illustrated with the help of a single elementliquid-rocket injector design problem in Section 6. Finally,Section 7 summarizes the main conclusions of the paper.

2. Basic terminology

This section defines some of the terms used in this paper.More precise definitions of the terms can be found in Refs.[1,5,39,40].

2.1. Multi-objective optimization and Pareto optimality

2.1.1. Search space

Search space or design space is the set of all possiblecombinations of the design variables. If all design variablesare real, the design space is given as x 2 RN (N is the num-ber of design variables). The feasible domain S is the regionin design space where all constraints are satisfied.

2.1.2. Multi-objective optimization problem formulation

Multi-objective optimization problem is formulated as

Minimize FðxÞ; where F¼ fj : 8j¼ 1;M ; x¼ xi : 8i¼ 1;N

Subject to:

CðxÞ6 0; where C¼ cp : 8p¼ 1; P ;

HðxÞ¼ 0; where H¼ hk : 8k¼ 1; K:

ð1Þ
2.1.3. Domination criteria
A feasible design x(1) dominates another feasible designx(2) (denoted as x(1) < x(2)), if both of the following condi-tions are true:

The design x(1) is no worse than x(2) in all objectives, i.e.,fj (x(1)) k fj (x(2)) for all j = 1,2, . . . ,M objectives

xð1Þ¥ xð2Þ ) 8j 2 Mfjðxð1ÞÞ� fjðxð2ÞÞ or

8j 2 Mfjðxð1ÞÞ 6 fjðxð2ÞÞ: ð2Þ

The design x(1) is strictly better than x(2) in at least one objec-tive, or fj (x(1)) < fj (x(2)) for at least one j 2 {1, 2, . . . ,M}

xð1Þ < xð2Þ ) ^j 2 Mfjðxð1ÞÞ < fjðxð2ÞÞ: ð3Þ

2.1.4. Non-dominated solutions [41]

If two designs are compared, then the designs are non-dominated with respect to each other if neither designdominates the other.

A design x 2 S (S is the set of all feasible designs) isnon-dominated with respect to a set A � S, if 9= a 2A:a < x.

Such designs in function space are called non-dominatedsolutions. Moreover, any design x is Pareto optimal if x isnon-dominated with respect to S.

2.1.5. Pareto optimal set

All the designs x (x 2 S) which are non-dominated withrespect to any other design in set S, comprise a set knownas Pareto optimal set.

2.1.6. Pareto optimal front (POF)The function space representation of the Pareto optimal

set is the Pareto optimal front. When there are two objec-tives, the Pareto optimal front is a curve, when there arethree objectives, the Pareto optimal front is representedby a surface and if there are more than three objectives,it is represented by a hyper-surface.

2.2. Response surface approximation

In this study, response surface approximations areemployed in two aspects, namely, to generate surrogates ofthe computationally expensive simulation-based modelsand to approximate the Pareto optimal front by representingone objective in terms of others. A brief description of theresponse surface methodology [39] is given in Appendix A.

3. Elitist non-dominated sorting genetic algorithm

(NSGA-II)

One of the multi-objective evolutionary algorithms(MOEA) that has been effective in finding the Pareto opti-mal solutions is the elitist non-dominated sorting geneticalgorithm (NSGA-II) developed by Deb et al. [8]. The algo-rithm is described as follows:

1. Randomly initialize population (designs in the variablespace) of size npop.

2. Compute objectives and constraints for each design.3. Rank the population using non-domination criteria

(many individuals can have same rank and rank-1 isthe best).

4. Compute crowding distance (this distance finds the rela-tive closeness of a solution to other solutions in the func-tion space and is used to differentiate between thesolutions on same rank).

5. Employ genetic operators – selection, crossover andmutation – to create intermediate population of sizenpop.

6. Evaluate objectives and constraints for this intermediatepopulation.

7. Combine the two (parent and intermediate) populations,rank them and compute the crowding distance.

8. Select new population of npop best individuals based onthe rank and crowding distance.

9. Go to step 3 and repeat till termination criteria isreached, which in the current study is chosen to be thenumber of generations.

In this study, the real-coded version of NSGA-II wasused that is, crossover and mutation operations were con-ducted in the real space rather than the binary space. For

Fig. 1. Demonstration of Pareto-drift problem: solutions at last genera-tion of NSGA-II simulation (reported Pareto optimal front) vs. allevaluated solutions. (For zoom in region, see Fig. 2.)

Fig. 2. Pareto-drift problem: loss of Pareto optimal solutions duringNSGA-II simulations (zoomed in view). Full Pareto optimal front isshown in Fig. 1.


all simulations a tournament selection operator with tour-nament size of two was used. An extensive parametric studywas conducted to select the parameters used in NSGA-IIalgorithm [42]. Based on the above mentioned studies, fol-lowing parameters were set for the simulations:

Population size (npop) 100Generations 250Crossover probability (Pcross) 1.00Distribution parameter (for crossover) 20Mutation probability (Pmut) 0.25Distribution parameter (for mutation) 200

4. Pareto drift in MOEAs

Most of the popular multi-objective evolutionary algo-rithms (MOEAs) are based on dominance criteria. Thesealgorithms investigate the POF using a finite populationsize and implement diversity preserving mechanisms suchas niching, clustering, crowding distances etc. [5, Chapters5–6] to find the complete POF. The solutions obtained ateach generation are characterized into dominated andnon-dominated solutions. These non-dominated solutionsare non-dominated with respect to the current set of solu-tions and may include the Pareto optimal solutions as wellas sub-optimal solutions. In non-elitist MOEAs, the geneticoperators may destroy some of these solutions to explorethe design space. Introducing elitism in MOEAs alleviatesthis problem to some extent, but when the number ofnon-dominated solutions in the combined populationexceeds the population size, as happens commonly in elitistMOEAs, some of the non-dominated solutions have to bedropped. If the solution thus lost is Pareto optimal, it maynot be recovered during the course of the optimization anda suboptimal solution can appear to be a non-dominatedsolution. This problem of losing Pareto optimal solutionsis defined as Pareto drift.

Fig. 1 shows one instance of this behavior for a two-objective optimization problem while using NSGA-II [8].All the solutions evaluated so far are shown by the dotsand the non-dominated solutions at the final generationof a NSGA-II simulation are shown by asterisks. Most ofthe solutions obtained after the final generation of aNSGA-II simulation are apparently non-dominated withrespect to all the evaluated solutions, but a few solutionsin the population at the final generation were dominated.This is clear in the zoomed view in Fig. 2, where most ofthe final generation solutions were dominated by the solu-tions evaluated during the search. This problem is commonto all dominance based methods.

One remedy to this problem is to maintain andcontinuously update an (unbounded-sized) archive of allthe non-dominated solutions obtained so far, similar tosome evolutionary strategies (e.g., PAES [7]) which main-tain a bounded-sized archive. The NSGA-II algorithm isaugmented with the archiving strategy and referred as

archiving NSGA-II (NSGA-IIa). The implementation canbe summarized as follows:

1. Initialize an archive with all the non-dominated solu-tions after Step 3 in the NSGA-II algorithm.

2. After Step 7 in the NSGA-II algorithm, update thearchive as follows:• Compare archive solutions with rank-1 solutions in

the combined population.• Remove all dominated solutions from the archive.• Add all rank-1 solutions in the current population

which are non-dominated with respect to the archive.

The final Pareto optimal set is the archive solutions set.This archiving of the Pareto optimal solutions retains the

Perform correlation check

Evaluate F, H, Cfm = f(x1, x2,…, xn)cp = c(x1, x2,…, xn)hk = h(x1, x2,…, xn)

Optimization problem Min F={f1, f2,…,fm}

fi = f(X)s. t. C(X) 0

H(X) = 0

Global Pareto optimal response surface

F* = F*(f1, f2, …, fm)

x1

x2

…xn

FHC

Generate Paretosolutions

f1f3

f2

Identify bounds on F* usingconvex hulls Clustering

f1f3

f2

Perform correlation check

Evaluate F, H, Cfm = f(x1, x2,…, xn)cp = c(x1, x2,…, xn)hk = h(x1, x2,…, xn)

Optimization problem Min F={f1, f2,…,fm}

fi = f(X)s. t. C(X) ≤ 0

H(X) = 0

Global Pareto optimal response surface

F* = F*(f1, f2, …, fm)

x1

x2

…xn

FHC

x1

x2

…xn

FHC


f1f3

f2


f1f3

f2

Identify bounds on F* usingconvex hulls Clustering

f1f3

f2

Clustering

f1f3

f2

Response surface method

Fig. 3. Flowchart of proposed method of constructing Pareto optimal front response surface approximation.


information of the good solutions obtained so far andimproves the convergence to the Pareto optimal front.On the negative side, the time and memory requirementsincrease substantially due to continuous increase in thearchive size and the need to compare all the archive solu-tions with the current generation non-dominated solutions.

5. Generating the Pareto optimal front response surface

approximation

Fig. 3 illustrates the methodology used for generatingthe Pareto optimal front RSA. The starting point is theidentification of design variables and their allowableranges, performance criteria and constraints. Once theproblem is defined, the designs are evaluated (objectivesand constraints) through an experiment or numerical sim-ulation as required by the problem. Because the cost ofdesign evaluation is often very high, computationally inex-pensive surrogate models are developed for both objectivesand constraints. Polynomial response surface approxima-tions were used in the context of this work. On the otherhand, for multi-objective problem, identifying correlatedobjectives may help reduce the complexity of the designproblem. Highly correlated objectives may be dropped3

or a representative objective can be used for all the corre-lated objectives (principal component analysis) hencereducing the dimensionality of the problem [42]. Afterdefining the multi-objective optimization problem, the Par-eto optimal solutions are generated. In this paper, aMOEA based hybrid method (NSGA-IIa + e-constraintstrategy [4], Appendix B) [42] is suggested as the multi-objective optimizer to generate Pareto optimal solutions.

The Pareto optimal front is then approximated by aresponse surface fit to the available Pareto optimal solu-tions. Since the quality of the response surface approxima-

3 Only the objectives which are strongly correlated to another should bedropped to reduce the dimensionality. If the objectives are weaklycorrelated, dropping objectives will cause loss of information.

tion depends on the number of data points, it is imperativeto generate a large number of the Pareto optimal solutionsto ascertain good quality. The Pareto optimal front RSArepresents one objective as a function of the other objec-tives, so it is important to choose this objective judiciously.Also note that the equation developed for the Pareto opti-mal front RSA is valid only in a limited region of functionspace. To identify the region where this response surfacerepresents the Pareto optimal front, convex hull(s) is (are)fitted to the approximating data [44]. When the responsesurface approximation to Pareto optimal solutions resultsin non-convex Pareto optimal front, identification of theboundary of the Pareto optimal front RSA using a singleconvex hull, may not be appropriate. Then it is more aptto fit several convex hulls to the subsets of Pareto optimalsolutions such that the non-convex boundary of the Paretooptimal front RSA is properly approximated. The multiplesubsets (clusters) of the Pareto optimal set can be effectivelyidentified using clustering. Details of the particular cluster-ing method used here are given in Appendix C. It is impor-tant to note that a sufficient number of subsets (clusters)should be selected to adequately capture the non-convexboundary.

This Pareto optimal front RSA shows the interactions ofthe different objectives for the global optimal values andhelps understand the physics of the problem. Once theclose form solution is available, the designer can visualizeand assess trade-offs among different objectives. This infor-mation can be used to refine the utility functions (impor-tance associated with different objectives) to come upwith more useful and practical designs.

6. An application: liquid-rocket single element

injector design

The proposed methodology was applied to design a sin-gle element injector of liquid-rocket engine. A schematicdiagram of the hybrid Boeing element injector under con-sideration is shown in Fig. 4.

Fig. 4. Schematic of hybrid Boeing element injector (US patent 6,253,539) and design variables.


6.1. Problem modeling

The injector design has two primary objectives:improvement of performance and life. As discussed byVaidyanathan et al. [20], the performance of the injectoris indicated by the axial length of the thrust chamber, whilethe survivability of the injector is associated with the ther-mal field inside the thrust chamber. A visual representationof the objectives is shown in Fig. 5. High temperaturesinduce high thermal stresses on the injector and the thrustchamber and thus reduce the life of the components butimprove the performance of the injector. Consequently,the objectives under consideration are:

1. Combustion length (Xcc) is the distance from the inlet,where 99% of the combustion is complete. It is desirableto keep the combustion length as small as possible asthis directly affects the size and efficiency of thecombustor.

2. Face temperature (TFmax) is the maximum temperatureof the injector face. It is desirable to reduce temperatureto increase the life of the injector.

3. Wall temperature (TW4) is the wall temperature at 3 in.(fourth probe) from the injector face. Higher values ofthe wall temperature reduce the life of the injector, sothis objective is minimized.

4. Tip temperature (TTmax) is the maximum temperatureon the post tip of the injector. It is desirable to keep thistemperature low to maximize life.

Fig. 5. Performance measures in single element liquid-rocket injector.

It can be seen that the dual goal of maximizing the per-formance and the life is now cast as a four-objective designproblem. As discussed by Vaidyanathan et al. [20] theseobjectives pose different and some times conflicting require-ments on the design scenarios, hence there cannot be a sin-gle optimal solution for this problem.

There are four design variables for the injector designproblem shown in Fig. 4. These design variables, theirbaseline values, and ranges are given as follows.

1. Hydrogen flow angle (a) the maximum angle variesbetween 0� to 20�. The baseline hydrogen flow angle is10�.

2. Hydrogen area (DHA) the increment with respect to thebaseline cross-section area (0.0186 in2) of the tube carry-ing hydrogen. The increment varies from 0% to 25% ofthe baseline hydrogen area.

3. Oxygen area (DOA) the decrement with respect to thebaseline cross-section area (0.0423 in2) of the tube carry-ing oxygen. The area varies between 0% and (�40)% ofthe baseline area.

4. Oxidizer post tip thickness (OPTT) varies between X00

to 2X00. The baseline value of tip thickness X00 is0.01 in.

All the variables were linearly normalized between 0 and1. More information about the design variables can befound in the work by Vaidyanathan et al. [20].

The boundary conditions applied in all CFD simula-tions are as follows. Both fuel and oxidizer flow in, throughthe inlet (west) boundary where the mass flow rate is fixedfor both streams. The nozzle exit, at the east boundary, ismodeled by an outlet boundary condition. The southboundary is modeled with the symmetry condition. Allwalls (both sides of the oxygen post, the outside of the fuelannulus, the outside chamber wall, and the faceplate) aremodeled with the no slip adiabatic wall boundary condi-tion. This is a computationally expensive simulation-basedproblem so for optimization purposes it is advisable todevelop surrogate models for the objective functions. Itwas shown by Vaidyanathan et al. [19,20], Shyy et al.[21], and Papila et al. [22,23] that accurate response sur-faces for complex problems like single element injectorscan be developed.

Table 1Accuracy of response surface approximation of objectives (refer toAppendix A for definition of accuracy measures)

TFmax Xcc TW4 TTmax

# of observations 38 38 38 52R2

adj 1.000 0.995 1.000 0.989ra 0.00566 0.0205 0.00803 0.0303Mean 0.495 0.497 0.514 0.591r (14 pts) 0.00460 0.0178 0.00669 –PRESS – – – 0.0388

4 Choice of dropping TW4 instead of TFmax was arbitrary since thecorrelation was very strong. Principal component analysis to identify threeorthogonal vectors would have been more appropriate.


6.2. Response surface approximation of objective functions

For constructing the surrogate models, Vaidyanathanet al. [20] conducted a design of experiments (orthogonalarrays [46]) to pick the design data points. CFD simula-tions were carried out for 54 data points. Simulations fortwo designs failed to produce valid results. Response sur-faces for all the objectives but TTmax were approximatedin normalized variable space using 38 design data pointsand tested using the remaining 14 designs. For the objectiveTTmax all 52 designs were used to fit the response surface.The response surfaces were fit using standard least-squaresregression with a quadratic polynomial using JMP [47]. Areduced cubic response surface was fitted to the objectiveTTmax and this was cross-validated using PRESS [39].Vaidyanathan et al. [20] obtained the following relationsbetween design variables and objective functions

TFmax ¼ 0:692þ 0:477ðaÞ � 0:687ðDHAÞ � 0:080ðDOAÞ� 0:0650ðOPTTÞ � 0:167ðaÞ2 � 0:0129ðDHAÞðaÞþ 0:0796ðDHAÞ2 � 0:0634ðDOAÞðaÞ� 0:0257ðDOAÞðDHAÞ þ 0:0877ðDOAÞ2

� 0:0521ðOPTTÞðaÞ þ 0:00156ðOPTTÞðDHAÞþ 0:00198ðOPTTÞðDOAÞ þ 0:0184ðOPTTÞ2;

ð4ÞX cc ¼ 0:153� 0:322ðaÞ þ 0:396ðDHAÞ þ 0:424ðDOAÞ

þ 0:0226ðOPTTÞ þ 0:175ðaÞ2 þ 0:0185ðDHAÞðaÞ� 0:0701ðDHAÞ2 � 0:251ðDOAÞðaÞþ 0:179ðDOAÞðDHAÞ þ 0:0150ðDOAÞ2

þ 0:0134ðOPTTÞðaÞ þ 0:0296ðOPTTÞðDHAÞþ 0:0752ðOPTTÞðDOAÞ þ 0:0192ðOPTTÞ2; ð5Þ

TW4 ¼ 0:758þ 0:358ðaÞ � 0:807ðDHAÞ þ 0:0925ðDOAÞ� 0:0468ðOPTTÞ � 0:172ðaÞ2 þ 0:0106ðDHAÞðaÞþ 0:0697ðDHAÞ2 � 0:146ðDOAÞðaÞ� 0:0416ðDOAÞðDHAÞ þ 0:102ðDOAÞ2

� 0:0694ðOPTTÞðaÞ � 0:00503ðOPTTÞðDHAÞþ 0:0151ðOPTTÞðDOAÞ þ 0:0173ðOPTTÞ2; ð6Þ

TTmax ¼ 0:370� 0:205ðaÞ þ 0:0307ðDHAÞ þ 0:108ðDOAÞþ 1:019ðOPTTÞ � 0:135ðaÞ2 þ 0:0141ðDHAÞðaÞþ 0:0998ðDHAÞ2 þ 0:208ðDOAÞðaÞ� 0:0301ðDOAÞðDHAÞ � 0:226ðDOAÞ2

þ 0:353ðOPTTÞðaÞ � 0:0497ðOPTTÞðDOAÞ� 0:423ðOPTTÞ2 þ 0:202ðDHAÞðaÞ2

� 0:281ðDOAÞðaÞ2 � 0:342ðDHAÞ2ðaÞ� 0:245ðDHAÞ2ðDOAÞ þ 0:281ðDOAÞ2ðDHAÞ� 0:184ðOPTTÞ2ðaÞ þ 0:281ðDHAÞðaÞðDOAÞ:

ð7Þ

The quality of the response surface approximations isgiven in Table 1 [20]. The response surfaces for all theobjectives had very high adjusted coefficient of multipledetermination which indicate good prediction capabilities.The rms error on the training and the test data points forresponse surfaces fitted to objectives TFmax and TW4 werevery low while the errors for the objectives Xcc and TTmax

(PRESS error instead of rms error) were relatively higherbut the values were reasonable.

For the chosen application, side constraints do notrequire any modeling; however, in a general case whenclosed form solution of objectives and constraints is notavailable, one can develop surrogate models for bothobjectives and constraints and use these surrogates in opti-mization as was done by Mack et al. [48] for design ofradial turbine used in liquid-rocket engine.

6.3. Define multi-objective optimization problem

Using correlation analysis, Goel et al. [42] found thatthe objectives TFmax and TW4 were strongly correlated inthe design space (Appendix D). Hence, TW4 was droppedfrom the objectives list4 and a multi-objective optimizationproblem was formulated with the remaining three objec-tives TFmax, Xcc, and TTmax (Eqs. (4), (5) and (7), respec-tively). The constraints on this problem were simplebounds on the variables (between 0 and 1).

6.4. Generate Pareto optimal solutions

The three-objective optimization problem was solvedusing NSGA-IIa and e-constraint strategy. At the end ofa NSGA-IIa simulation, the total number of optimal solu-tions in the archive was 5724 and the population at the finalgeneration had 100 non-dominated solutions. For thisproblem, the archive corresponds to less than 350 KB ofmemory (using double precision type for all variables andfunctions) which is modest for current computational capa-bilities. This indicates that the size of archive may not be acritical issue in terms of memory requirements for complexproblems with more objectives, variables and constraints.Future improvements in computer hardware will further


render this issue of memory size trivial. The computationalexpense of comparing and replacing non-dominating solu-tions in the archive can be reduced by implementing effi-cient search algorithms.

A comparison of the solutions in archive with the solu-tions at the final iteration is shown in Fig. 6. The majority(64 out of 100) of the non-dominated solutions at final iter-ation were dominated by the optimal solutions in thearchive. To improve robustness of the results, the NSGA-IIa simulations were repeated with ten random initial seeds.Results from the NSGA-IIa and the NSGA-II werecompared for each simulation. It was observed that onan average 63 out of 100 (range = [53, 68], standard devia-tion = 5) non-dominated solutions in the NSGA-II finaliteration were dominated by the corresponding archivesolutions. This demonstrates the effectiveness of the archiv-ing strategy in preventing the loss of potential Pareto opti-mal solutions.

To further improve the convergence to the Pareto opti-mal front, the archives from the ten NSGA-IIa simulationswere combined. The dominated and duplicate designs wereremoved from the combined archive to get 29,650candidate optimal designs. To assess the improvementsby using multiple simulations, each archive from theNSGA-IIa simulation was compared with the combinedarchive. It was observed that on an average 2346 solutions

Table 2Average and maximum improvements in solution quality (distance between darchiving, (b) multiple simulations and (c) hybridization (MOEA + local searc

Average improvement

Mean* Std dev.* Min M

Archiving 0.156 0.039 0.090 0Multiple simulations 0.050 0.015 0.032 0Hybridization 0.008

*Based on 10 simulations.

Fig. 6. Comparison of the solutions in the archive with the solutions atfinal iteration (it demonstrates that the archive alleviates the Pareto drift).This result is shown for one representative NSGA-II/a simulation.

(range = [2169,2620], standard deviation = 137) from theindividual archives were dominated by the combinedarchive of the solutions.

Goel et al. [42] demonstrated that local search using e-constraint strategy can further improve the solutionsobtained using MOEA. Following the procedure outlinedin Appendix B, 118,600 solutions were obtained in acombined pool of solutions from local search and theNSGA-IIa simulations. There were 88,553 non-dominatedsolutions. After removing the duplicates, 87,149 Paretooptimal solutions were found. This final solution set dom-inated 29,483 solutions out of 29,650 solutions from thecombined pool of the solutions obtained after multipleNSGA-IIa simulations. The results manifest the improve-ments using the hybrid approach of using MOEA and localsearch [42].

The improvements in the solutions quality (given as thedistance between the dominated solution and the solutionwhich dominates this solution) from each step (archiving,multiple simulations and hybridization) are quantified bycomputing average and maximum improvements and aretabulated in Table 2. The results of comparison betweenthe NSGA-II and the NSGA-IIa (archiving), and betweenindividual archive and combined archive (multiple simula-tions) are averaged over 10 simulations. For all steps, theaverage improvements were much smaller than the maxi-mum improvements. The effect of subsequent steps on con-vergence to the Pareto optimal front (the average andmaximum improvements in the solution quality) reducedwhich demonstrated the convergence to the Pareto optimalfront. Relatively smaller reduction in maximum improve-ments demonstrates that different steps are effective in con-verging the far-from-optimal solutions to Pareto optimalfront.

To elucidate the impact of this three step method toachieve converged Pareto optimal solution, a simpleexample is presented as follows. Suppose a designer decidesto give equal importance to all objectives (typicalpreference structure), the objective function f to min-imize is TFmax + Xcc + TTmax. If she uses only NSGA-IIsimulations results, the best solution obtained is f =0.8106 (TFmax = 0.3790, Xcc = 0.3431, TTmax = 0.0885).After implementing the archiving strategy, her bestresult is f = 0.7692 (TFmax = 0.3811, Xcc = 0.3518,TTmax = 0.0363). With multiple simulations her resultimproves to f = 0.7670 (TFmax = 0.3819, Xcc = 0.3503,

ominated solution and the solution which dominates this solution) by (a)h)

Maximum improvement

ax Mean* Std dev.* Min Max

.249 0.560 0.073 0.436 0.690

.083 0.466 0.041 0.415 0.5270.324

Table 3Objective functions and design variables for nine representative Paretooptimal designs (refer to Figs. 4 and 5)

a DHA DOA OPTT TFmax Xcc TTmax

0.000 1.000 0.629 0.803 0.015 0.957 0.9370.000 1.000 0.202 0.669 0.033 0.655 0.9550.000 1.000 0.308 0.281 0.044 0.689 0.7030.017 1.000 0.897 0.000 0.067 1.023 0.3930.246 1.000 0.163 0.000 0.171 0.503 0.3650.624 0.665 0.000 0.000 0.498 0.260 0.2180.300 0.104 0.000 0.260 0.730 0.122 0.5620.472 0.090 0.000 0.000 0.818 0.076 0.2501.000 0.018 0.562 0.000 0.936 0.117 �0.013


TTmax = 0.0348) and after hybridization, her final solutionis f = 0.7667 (TFmax = 0.3817, Xcc = 0.3504, TTmax =0.0346). Solving this single objective optimization problemusing sequential quadratic programming gives the sameresult as the final solution.

6.5. Representative designs

The procedure discussed previously resulted in a Paretooptimal solution set with 87,149 solutions. In order to illus-trate alternative representative design concepts, a hierarchi-cal clustering algorithm (given in Appendix C) was used inthe function space. Nine compromise solutions wereselected. These solutions vis-a-vis Pareto optimal solutionset are shown in Fig. 7. Fig. 7 shows that the representativesolutions were uniformly selected from the Pareto optimalfront. It was observed that small values of the face temper-ature TFmax in general were accompanied by longer com-bustion length Xcc and higher tip temperature TTmax.However, a small compromise in the face temperaturecan substantially reduce the combustion length and tiptemperature. Also it was observed that the combustionlength can be substantially reduced by allowing a high ther-mal environment in the combustion chamber. Selecteddesigns represent these distinct regions.

Objective function values and design parameter valuesfor the nine representative designs are given in Table 3.Some interesting results about the physics of the problemwere observed from Table 3. In general, the smaller valuesof the flow angle yield low face temperature, but highertemperature on the injector tip and longer combustionlength. Increase in the flow angle of the injector causesan increase in the temperature of the face of injector anda reduction in the length of the combustor. It is also seenthat if the cross-section area of the oxygen tube is reduced,for low values of flow angle and high value of cross-sectionarea of hydrogen tube, the temperature at the face increasesand the combustion length reduces. There is a strong effectof cross-interactions among different variables, which

Fig. 7. Nine representative trade-off solutions obtained using clustering.

impact the optimal design. This does not allow drawingconclusion about the effect of individual parameters. Amore detailed analysis of the results is given by Vaidyana-than et al. [49].

6.6. Pareto optimal front response surface approximation

Visualization of the Pareto optimal front in three-dimensions is cumbersome and only qualitative inferencesabout different design domains and function trade-offscan be made. This problem is alleviated by approximatingthe Pareto optimal front with a polynomial-based responsesurface. All 87,149 Pareto optimal solutions were used to fitthe response surface. It was unknown which of the objec-tives can be represented as a function of the remainingobjectives more accurately. Hence, each objective functionwas represented in terms of the other two objective func-tions considering different order polynomials. Since thecost of fitting a polynomial response surface model is verysmall compared to the cost of identifying Pareto optimalsolutions, it is recommended to try all (Nobj) combinationsbefore selecting the final form of relationship betweenobjectives.

The quality indicators of different response surfaceapproximations are presented in Table 4. Several observa-tions can be made from Table 4. First, choosing the properobjective to be represented as a function of the remainingtwo objectives influences the quality of the approximationsubstantially. In this case, TFmax = TFmax(Xcc, TTmax)gave the best quality of the response surface and TTmax =TTmax(TFmax, Xcc) gave the worst results. Second, asexpected, increasing the order of the polynomial gives abetter fit. With increasing order of the polynomial, R2

adj

increased and rms error and number of points with higherrors (>0.1) reduced. The maximum error for Xcc =Xcc(TFmax, TTmax) reduced with increase in the order ofpolynomial but the effect was unclear for the remainingtwo approximations. Finally, a quintic response surfaceapproximation of TFmax = TFmax(Xcc, TTmax) was selectedas Pareto optimal front RSA. The R2

adj was very good. ThePareto optimal front RSA fits the data very well. The coef-ficients of the Pareto optimal front RSA and the corre-sponding t-statistics are given in Table 5. High values of

Table 4Accuracy of response surface approximations of the 87,149 Pareto optimal solutions (refer to Appendix A for definition of accuracy measures)

TFmax = f(Xcc, TTmax) Xcc = f(TFmax, TTmax) TTmax = f(TFmax, Xcc)

Mean response 0.475 0.357 0.369Quadratic R2

adj 0.977 0.883 0.754RMS error 0.0560 0.0980 0.161Max error 0.236 0.400 0.452# of points with error > 0.1 5512 25,840 55,157

Cubic R2adj 0.984 0.926 0.899

RMS error 0.0462 0.0783 0.103Max error 0.210 0.356 0.298# of points with error > 0.1 4001 18,762 27,287

Quartic R2adj 0.991 0.969 0.937

RMS error 0.0349 0.0506 0.0815Max error 0.245 0.300 0.555# of points with error > 0.1 1974 4964 17,712

Quintic R2adj 0.993 0.983 0.956

RMS error 0.0316 0.0371 0.0679Max error 0.262 0.280 0.273# of points with error > 0.1 1304 1806 11,967


t-statistics for all the terms showed statistical significanceof the coefficients.

The contour plot of the Pareto optimal front RSATFmax = TFmax(Xcc, TTmax) along with the Pareto optimalsolutions illustrates its application domain in Fig. 8. It isimportant to note that Table 5 represents the Pareto opti-mal front for limited ranges of Xcc and TTmax. Only theregions bounded by the Pareto optimal solutions shown

Table 5Coefficients of response surface representing Pareto optimal frontTFmax = TFmax(Xcc, TTmax)

Term Coefficient t-ratio

Intercept 1.03 1632.7Xcc 0.27 23.5TTmax �1.09 �127.1X 2

cc �13.34 �166.6XccTTmax �0.62 68.15

TT2max 2.53 �10.07

X 3cc 33.41 130.4

X 2ccTTmax 2.99 �12.2

X ccTT2max �4.47 10.84

TT3max �0.95 �23.22

X 4cc �30.74 �90.15

X 3ccTTmax �2.38 �5.22

X 2ccTT2

max �2.68 �7.09

X ccTT3max 8.46 23.77

TT4max �2.17 �16.6

X 5cc 10.84 79.23

X 4ccTTmax �5.57 �13.08

X 3ccTT2

max 14.32 16.53

X 2ccTT3

max �12.16 �14.29

X ccTT4max 1.76 3.75

TT5max 0.58 5.33

Fig. 8. Contour plot of the Pareto optimal front response surfaceapproximation TFmax = TFmax(Xcc, TTmax) with Pareto optimal solutions.

as dots in Fig. 8 represent the Pareto optimal front. Thismeans that the zones with extreme values of the objectiveTFmax (>1.00 or <0.00) are not part of the Pareto optimalfront. The analysis of the response surface contours tellsthe designer that TFmax decreases with increase in Xcc

and TTmax. For every value of Xcc an optimal value ofTTmax can be found such that TFmax is minimized andvice-versa. For Xcc < 0.40, increase in Xcc is more effectivein reducing TFmax as compared to the increase in TTmax.This trend is reversed in the region Xcc > 0.40. The insightgained in the present investigation has further elucidatedthe interactions of the objectives in specific ranges, besidesthose observed by Vaidyanathan et al. [20].

It is also evident from Fig. 8 that for a good compromiseamong all the objectives, the values of normalized Xcc

should be between 0.35–0.60 and TTmax should be between

Table 6Data points in function space defining the boundary of one representativeconvex hull

Xcc TTmax TFmax

0.093 0.937 0.3870.108 0.937 0.3800.098 1.035 0.3840.069 1.094 0.3880.066 1.094 0.3890.055 0.951 0.4050.055 0.942 0.4060.055 0.940 0.4060.073 0.938 0.3970.093 0.937 0.387


0.00 and 0.30 of its normalized value. For this range ofvalues, TFmax varies between 0.10 and 0.40 of its normal-ized value. This information is very useful for a designer,in selecting the trade-offs among all the objectives orimproving the already existent preference structure. Forexample, the optimization with equal importance to allthe objectives had led to a design with TFmax = 0.3817,Xcc = 0.3504, TTmax = 0.0346. But following the informa-tion about the Pareto optimal front RSA, he may allowan increase in objective TTmax to reduce the other twoobjectives. Once the close form relation is available to thedesigner relating different objectives, a sensitivity analysiscan be done to find a more exact quantification of thetrade-offs for the selected design. At the optimal designwhen all objectives have equal importance, the derivativesusing the response surface are given as oTFmax/oXcc = �1.26; oTFmax/oTTmax = �1.04. This indicates thatincreasing Xcc by a unit amount keeping TTmax constant ismore (�21%) effective in reducing TFmax than reducingTTmax while keeping Xcc constant. However, since objec-tives Xcc and TTmax are not independent, one must ensurethat changes in the design objectives do not violate theboundary of the Pareto optimal front.

The boundary of the application regions of this non-convex response surface was identified using multiple con-vex hulls of the Pareto optimal solutions. One convex hullwas identified in each cluster of points. The convex hullidentifies the set of points on the boundary such that allthe points in the cluster are within the bounded region.For this problem, convex hulls using 9 clusters were notsufficient to represent the boundary of the Pareto optimalresponse surface. A significant non-Pareto optimal regionwas bounded by the convex hulls. Hence convex hulls werefitted to the data from 30 clusters. These 30 clusters, cent-roids of the clusters and the corresponding bounding con-

Fig. 9. Convex-hulls to bound the domain where the Pareto optimal frontRSA TFmax = TFmax(Xcc, TTmax) is valid. Dots show the centers of theclusters for each convex hull and thick lines show the boundary of eachcluster.

vex hulls are shown on the contour plot of the responsesurface in Fig. 9. The points defining the boundary ofone representative convex hull are given in Table 6. Theboundary of the response surface is identified adequatelywith the convex hulls in most of the regions, but there weresome small regions where the convex hulls bounded non-Pareto optimal front. The non-Pareto optimal front regioncan be further reduced by using a large number of clusters;however there is a trade-off between the number of clusters(and convex hulls) and the non-Pareto optimal front regionfor non-convex problems. In order to select a combinationof Xcc and TTmax, where the Pareto optimal front RSA isvalid, the selected point must be bounded by at least oneconvex hull.

6.7. Comparison of 3-objective vs. 4-objective optimization

problem results

To verify the effect of reducing the objectives, a singleNSGA-IIa simulation based hybrid strategy was used tosolve optimization problems with all 4-objectives and 3-objectives, which resulted in 18,812 and 16,467 uniquenon-dominated solutions. These two non-dominated solu-tion sets were compared. Most of the solutions obtainedwere non-dominated with respect to one another. Foursolutions from the 4-objective NSGA-IIa simulation resultswere dominated by solutions obtained for the 3-objectiveoptimization, and eight solutions obtained from the 3-objective simulation were dominated by the 4-objectivesimulation results. The number of dominated solutionsfor both populations was very small when the total numberof solutions is considered. The diversity of the solutions inobjective function space was also comparable for the twocases. This verifies that there were no adverse effects intro-duced by reducing the number of objectives.

7. Concluding remarks

The Pareto optimal front in multi-objective optimiza-tion problems is useful to visualize and assess trade-offsamong different design objectives. In addition to identifycompromise solutions, this also helps the designer set real-istic design goals. In this paper, a systematic approach for


providing an approximate closed form solution to the Par-eto optimal front and associated issues are presented. Thisapproach comprise of the following steps: (i) evaluation ofdesign by direct simulation or surrogate models, (ii) deter-mination of the Pareto optimal solutions by multi-objectiveoptimizer, (iii) approximation of Pareto optimal frontusing response surface and (iv) approximation of theboundary using convex-hull(s) of the Pareto optimal solu-tions data. The selection of the objective to be representedas a function of the other objectives has a great bearing onthe accuracy of fit of the Pareto optimal front RSA andhence should be carefully selected by trying the differentpossibilities. The Pareto optimal front RSA does not repre-sent the Pareto front for all combinations of the objectivesso the relevant domain should be properly bounded. Con-vex hulls fit to the Pareto optimal solutions data can beused to locate the boundary of the domain where theresponse surface represents the Pareto optimal front. Fora non-convex Pareto optimal front, the bounds should beidentified using convex hulls fitted to the multiple clustersof the Pareto optimal solutions. The centers of the clusterscharacterize alternative optimal design concepts which canbe selected for further assessment.

The problem of Pareto drift (losing Pareto optimal solu-tions during course of optimization) in the dominancebased MOEAs is identified. Implementation of an archiv-ing strategy to preserve all good solutions is suggested asa remedy. Specifically, NSGA-II algorithm is modified toimplement the archiving strategy and the improvementsin the quality of the solutions and convergence to Paretooptimal front were demonstrated. On the negative side,the computational cost of finding the Pareto optimal solu-tions has increased due to continuous update of thearchive.

The proposed approach is exemplified using a four-objective liquid-rocket single-element injector design prob-lem. The response surface method was employed in twoaspects. First, response surfaces based on the CFD solu-tions were used to evaluate the design objectives [20]. Thisresponse surface related the design variables and the objec-tives. Second, the Pareto optimal front was approximatedby response surface. This response surface representedone objective in terms of others objectives. The complexityof the multi-objective optimization problem was reducedby removing one (TW4) of the two correlated objectives(TFmax and TW4). The resulting multi-objective optimiza-tion problem was solved using a combination of NSGA-IIa and e-constraint strategy. The solution of the multi-objective optimization problem was the Pareto optimalsolutions data which represented a non-convex Paretofront. The Pareto optimal front was approximated by aquintic response surface TFmax = TFmax(Xcc, TTmax). Thedomain of application of this Pareto optimal front RSAwas bounded by convex hulls fitted to multiple clusters ofthe Pareto optimal solutions data.

The analysis of Pareto optimal solutions revealed impor-tant information about the effect of design variables on the

objectives. For example, smaller values of the flow angleyields low face temperature, but higher temperature oninjector tip and longer combustor length. Interactionsamong different objectives as identified by the Pareto opti-mal response surface manifested the importance of eachdesign objective in different regions of the function space.For example, the face temperature can be reduced moreeffectively by allowing small increase in combustion lengthfor small combustors, whereas allowing small rise in tiptemperature is more effective for moderate length combus-tors. The Pareto optimal response surface also helped inthe visualization of trade-offs among designs. It wasobserved that a good compromise region with equal impor-tance to all objectives, will have 0.35 < Xcc < 0.60, 0.00 <TTmax < 0.30 and 0.10 < TFmax < 0.40.

In summary, the proposed systematic approach wasfound useful when solving a complex multi-objectiveliquid-rocket injector design problem and holds promiseto be effective in broader contexts.

Acknowledgement

The present effort has been supported by the NASAConstellation University Institute Program (CUIP), Ms.Claudia Meyer program monitor.

Appendix A. Response surface methodology [39]

The response surface method fits a function to a set ofexperimentally or numerically evaluated design datapoints. There are various response surface approximationmethods available in the literature [39], with polynomial-based approximations being the most popular amongpractitioners. In this technique, an appropriate orderpolynomial is fitted to a set of data points, such that theadjusted rms (root mean square) error ra [39] is minimized.The adjusted rms error ra [39] is defined as following:

Let Np be the number of data points, Nc be the numberof coefficients, and error ei at any design point i beingdefined as

ei ¼ yi � yi; ð8Þ

where yi is the actual value of the function at the designpoint and yi is the predicted value. Hence

ra ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiXN

i¼1

e2i

,ðNp � NcÞ

vuut : ð9Þ

If Nt additional test data are used to test the quality of theapproximation, the rms error r is given as

r ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiXM

i¼1

e2i

,Nt

vuut : ð10Þ

Prediction capability of the response surface is given by thecoefficient of multiple determination R2

adj defined as,


R2adj¼ 1�r2

aðN p�1ÞXNp

i¼1

ðyi��yÞ2 where �y¼XNp

i¼1

yi

,N p

,:

ð11ÞFor a good fit, R2

a should be closer to 1.When the number of data points is not adequate enough

to spare enough data for testing the response surface (RS),PRESS statistic is used to estimate the performance of theRS. A residual is obtained by fitting a RS over the designspace after dropping one design point from the trainingset and then comparing the RS predicted value for thatpoint with the expected value. PRESS is the square sumof the residuals and is given by

PRESS ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiXNp

i¼1

yi � y�ið Þ2,

N p

vuut ð12Þ

where y�i is the value predicted by the RS for the ith pointwhich is excluded while generating the RS. If this value isclose to ra then the model performs well. More details onthe polynomial response surface approximation can befound in the classical text by Myers and Montgomery [39].

Appendix B. MOEA + local search strategy [42]

After running the evolutionary algorithm, a set ofapproximate Pareto optimal points was obtained. It wasshown by Goel and Deb [13–15], and Ishibuchi et al. [16]that using a local search method on this approximate Paretooptimal solution set can further improve the solutions andhelp better converge to the Pareto optimal set. There are anumber of different ways of using the local search in con-junction with EA. Goel and Deb [15] studied two extremesof hybridization, a posteriori hybrid method where localsearch starts from the EA simulation results, and an onlinehybrid method which uses local search after every genera-tion of the EA simulation. They found that the posteriorihybrid method performs better than the online hybridmethod in terms of convergence to the Pareto optimal frontand ensuring diversity on the Pareto optimal front. In thecurrent study, the posteriori hybrid method is used andthe sequential quadratic programming method implementa-tion in MATLAB [43] is used as the local optimizer.

When using a local search method, the multi-objectiveoptimization problem has to be converted to a single objec-tive optimization problem. The weighted sum strategy –where multiple objectives are combined in a single objectiveby associating a weight to each objective – is very popularand simple to implement. However, it was shown by Goelet al. [42] that this method does not always improve thesolutions. Alternatively, the e-constraint strategy givesgood results for two-objective optimization problems[1,40]. Goel et al. [42] extended this idea to the multi-objec-tive optimization problems with more than two objectives.The procedure of using the e-constraint strategy to improvethe convergence to Pareto optimal front is given as follows:

1. For any candidate optimal solution n, formulate con-strained single objective optimization problem

Minimize f jðxÞ; where x ¼ xi : 8i ¼ 1;N

Subject to:

CðxÞ 6 0; where C ¼ cp : 8p ¼ 1; P ;

HðxÞ ¼ 0; where H ¼ hk : 8k ¼ 1;K;

f mðxÞ 6 em 8m ¼ 1;M ; m 6¼ j; em ¼ f ðnÞm :

ð13Þ

Note that the last constraint was not present in the ori-ginal problem.2. Solve the constrained optimization problem given in

Eq. (13) for "j = 1,M using current design x(n) as initialguess.

3. Sequential quadratic programming algorithm in MAT-LAB� [43] was used in this paper.

4. If the optimal solution to this problem is found, thissolution is accepted else original solution is taken asoptimal solution.

5. Repeat Step 1 for all the candidate optimal solutions.6. For each candidate optimal solution, M new solutions

are obtained using local search.7. Combine all the optimal and candidate solutions in a set

SPOF.8. Remove sub-optimal solutions from the set SPOF by

performing a non-dominated search.9. Remove duplicate designs from set SPOF.

The solution set SPOF represents the Pareto optimal set.

Appendix C. Clustering algorithm

Clustering of the Pareto optimal solution set dataserves two purposes: first, to identify the subsets whichcan be used to identify the boundary of the Pareto opti-mal response surface and second, to reduce the largenumber of Pareto optimal solutions data to a manageablenumber of trade-off designs which can be evaluated by thedesigner. If Nclust designs are required, a clustering algo-rithm groups the entire solution set into Nclust distinctclusters and picks one representative solution from eachclusters. This representative solution is the closest to themean of the parent cluster. Each representative solutionof the cluster can be used by the designer to evaluatethe design and the solutions in each cluster representthe subset which can be used to find the boundary ofthe Pareto optimal response surface by fitting the convexhulls. The clustering procedure, similar to hierarchicalclustering method [45], followed in this paper is given asfollows:

1. Reduce the large number of Pareto optimal solutionsto a more manageable number by selecting thedesigns with high crowding distance. In this study,


3000 designs out of 89,147 designs with highestcrowding distance were selected.5

2. Since crowding distance favors lonely solutions, thisset of solutions used for clustering algorithm is aug-mented by selecting random solutions which favorthe regions with high density. Thus compensatingthe loss of the solutions in the crowded regions.Another 2000 designs were selected randomly.

3. Initialize all the solutions in this set as independentclusters.

4. Identify the power p such that npop � 2pNclust.5. For p/2 iterations,

a. Compute distances between each cluster.b. Merge all the neighboring clusters such that each

cluster is merged with only one cluster.c. Find the new centroid of the cluster.d. Repeat step 5a.This step merges many neighbors at one time, reduc-ing the cost of computation in clustering.

6. Now compute the distance between the centroids ofeach cluster.

7. Merge the two clusters with smallest distance betweenthe centroids.

8. Re-compute the centroid of the new cluster.9. Repeat step 6 till the number of clusters is reduced to

Nclust.10. Find the data point which was closest to the centroid

of the cluster.11. Now for each data point in the Pareto optimal set,

a. Compute the distance from the centroid of Nclust

clusters.b. Move the data point to the cluster whose centroid

is closest to the data point.

6

12. Repeat step 11 for all the points in the Pareto optimalset.

This method reduces the cost of clustering substantiallywhile identifying the representative solution of the clusterand the members in each cluster.

Appendix D. Reduced optimization problem after correlation

analysis [42]

Vaidyanathan et al. [20] pointed out that all four objec-tives, based on the designs obtained during their individualminimization largely fall into two groups: (i) TFmax andTW4 and (ii) Xcc and TTmax. Minimizing TFmax and TW4

leads to a design with a equal to zero (making the designa shear coaxial type), maximum fuel flow area and thickestpost tip. This design yields less desirable performance dueto slower mixing across the shear layer. Minimizing TTmax

and Xcc, results in an impinging-like design with a equal to

5 This step is required to limit the computational cost of clusteringoperation. If the number of Pareto optimal solutions is small, Steps 1 and2 can be ignored.

one. It also has the minimum fuel flow area and the thin-nest post tip thickness. This design performs well, but hasvery high wall and injector face temperatures.

Goel et al. [42] systematically investigated the correla-tions among the objectives in entire design space beforesolving the optimization problem. A uniform grid data(114 = 14,641 designs) was generated and evaluated usingthe response surface approximations for the objective func-tions. The correlation matrix Cdes and corresponding p-val-ues were computed6 using MATLAB� with propercorrections for the boundary points

Cdes ¼

TFmax X cc TW4 TTmax

TFmax 1:00 �0:12 0:96 0:28

X cc �0:12 1:00 �0:03 0:57

TW4 0:96 �0:03 1:00 0:30

TTmax 0:28 0:57 0:30 1:00

26666664

37777775: ð14Þ

The correlation matrix (Cdes) showed that the objectivefunctions TFmax and TW4 were strongly correlated in thedesign space. p-values and 95% confidence intervals forthe correlation coefficients also established the statisticalsignificance of the results. The objectives Xcc and TTmax

were found to be weakly correlated. These findings are inagreement with the results observed by Vaidyanathanet al. [20].

References

[1] K.M. Miettinen, Nonlinear Multiobjective Optimization, Kluwer,Boston, 1999.

[2] A. Messac, Physical programming: effective optimization for compu-tational design, AIAA J. 34 (1) (1996) 149–158.

[3] P. Sen, J.-B. Yang, Multiple Criteria Decision Support in EngineeringDesign, Springer-Verlag, London, 1998.

[4] V. Chankong, Y.Y. Haimes, Multiobjective Decision Making Theoryand Methodology, Elsevier Science, New York, 1983.

[5] K. Deb, Multi-objective Optimization Using Evolutionary Algo-rithms, Wiley, Chichester, UK, 2001.

[6] E. Zitzler, L. Thiele, An evolutionary algorithm for multi-objectiveoptimization: The strength Pareto approach, Technical report no. 43,Zurich: Computer engineering and networks laboratory Switzerland,1998.

[7] J. Knowles, D. Corne, Approximating the non-dominated front usingthe Pareto archived evolution strategy, Evol. Comput. 8 (2) (2000)149–172.

[8] K. Deb, S. Agrawal, A. Pratap, T. Meyarivan, A fast and elitist multi-objective genetic algorithm for multi-objective optimization: NSGA-II, in: Proceedings of the Parallel Problem Solving from Nature VIConference, Paris, 2000, pp. 849–858.

[9] K. Deb, T. Goel, Controlled elitist non-dominated sorting geneticalgorithms for better convergence, in: Proceedings of the FirstInternational Conference on Evolutionary Multi-criterion Optimiza-tion, Zurich, 2001, pp. 67–81.

Here quadrature is used to numerically compute correlations. One canalso resort to Monte-Carlo simulations or even analytical integrals tocompute correlations. However since the correlations are computed inentire design space, the error in computation of correlations should be nomore than small numerical error irrespective of the choice of samplingpoints.


[10] D. Sasaki, M. Morikawa, S. Obayashi, K. Nakahashi, Aerodynamicshape optimization of supersonic wings by adaptive range multi-objective genetic algorithms, in: Proceedings of the First InternationalConference on Evolutionary Multi-criterion Optimization, Zurich,2001, pp. 639–652.

[11] D. Sasaki, S. Obayashi, K. Nakahashi, Navier–Stokes optimizationof supersonic wings with four objectives using evolutionary algo-rithms, J. Aircraft 39 (4) (2002) 621–629.

[12] R.A.E. Makinen, J. Periaux, J. Toivanen, Multidisciplinary shapeoptimization in aerodynamics and electromagnetics using geneticalgorithms, Int. J. Numer. Methods Fluids 30 (1999) 149–159.

[13] K. Deb, T. Goel, Multi-objective evolutionary algorithms forengineering shape design, in: R. Sarker, M. Mohammadin, X. Yao(Eds.), Evolutionary Optimization, Kluwer, 2000, pp. 147–176.

[14] K. Deb, T. Goel, A hybrid multi-objective evolutionary approach toengineering shape design, in: Proceedings of Evolutionary Multi-criterion Optimization Conference, Zurich, 2001, pp. 385–399.

[15] T. Goel, K. Deb, Hybrid methods for multi-objective evolutionaryalgorithms, in: Proceedings of the 4th Asia-Pacific Conference onSimulated Evolution and Learning Computational Intelligence for thee-age SEAL’02, Singapore, 2002, pp. 188–192.

[16] H. Ishibuchi, T. Yoshida, T. Murata, Balance between genetic searchand local search in memetic algorithms for multi-objective permuta-tion flowshop scheduling, IEEE Trans. Evol. Comput. 7 (2) (2003)204–223.

[17] S. Obayashi, D. Sasaki, A. Oyama, Finding tradeoffs by using multi-objective optimization algorithms, Trans. Jpn. Soc. Aeronaut. SpaceSci. 47 (155) (2004) 51–58.

[18] J.I. Madsen, W. Shyy, R.T. Haftka, Response surface techniques fordiffuser shape optimization, AIAA J. 38 (2000) 1512–1518.

[19] R. Vaidyanathan, N. Papila, W. Shyy, K.P. Tucker, L.W. Griffin,R.T. Haftka, N. Fitz-Coy, Neural network and response surfacemethodology for rocket engine component optimization, in: 8thAIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Anal-ysis and Optimization, Paper No. 2000-4480, Long Beach, CA, 2000.

[20] R. Vaidyanathan, K.P. Tucker, N. Papila, W. Shyy, CFD baseddesign optimization for a single element rocket injector, J. PropulsionPower 20 (4) (2004) 705–717 (also presented in 41st AerospaceSciences Meeting and Exhibit, Reno, NV, 2003).

[21] W. Shyy, N. Papila, R. Vaidyanathan, K. Tucker, Global designoptimization for aerodynamics and rocket propulsion components,Progr. Aerospace Sci. 37 (2001) 59–118.

[22] N. Papila, W. Shyy, L.W. Griffin, F. Huber, K. Tran, Preliminarydesign optimization for a supersonic turbine for rocket propulsion, in:36th AIAA/ASME/SAE/ASEE Joint Propulsion Conference andExhibit, Huntsville, AL, 2000.

[23] N. Papila, W. Shyy, L.W. Griffin, D.J. Dorney, Shape optimization ofsupersonic turbines using response surface and neural networkmethods, J. Propulsion Power 18 (2001) 509–518.

[24] R. Dornberger, D. Buche, P. Stoll, Multidisciplinary optimization inturbomachinery design, in: European Congress on ComputationalMethods in Applied Sciences and Engineering, Barcelona, 2000.

[25] A. Bramanti, P. Di Barba, M.B. Farina, A. Savini, Combiningresponse surfaces and evolutionary strategies for multi-objectivePareto-optimization in electromagnetics, Int. J. Appl. Electromagn.Mech. 15 (2001) 231–236.

[26] B. Wilson, D. Cappelleri, T.W. Simpson, M. Frecker, Efficient Paretofrontier exploration using surrogate approximations, Optim. Engrg. 2(2001) 31–50.

[27] D.J. Cappelleri, M.I. Frecker, T.W. Simpson, A. Snyder, Ametamodel based approach for optimal design of a PZT bimorphactuator for minimally invasive surgery, ASME J. Mech. Des. 124 (2)(2002) 354–357.

[28] M. Farina, J.K. Sykulski, Comparative study of evolution strategiescombined with approximation techniques for practical electromag-

netic optimization problems, IEEE Trans. Magn. 37 (5) (2001) 3216–3220.

[29] M. Farina, P. Di Barba, A. Bramanti, A GRS method for Pareto-optimal front identification in electromagnetic multi-objective syn-thesis, IEEE Proc. Sci. Manage. Technol. 149 (5) (2002) 207–213.

[30] M. Emmerich, A. Giotis, M. Ozdenir, T. Baeck, K. Giannakoglou,Metamodel-assisted evolution strategies, in: Proceedings of ParallelProblem Solving from Nature (PPSN VII), Springer LNCS, 2002, pp.361–370.

[31] M. Emmerich, K. Giannakoglou, B. Naujoks, Single and multi-objective optimization assisted by Gaussian random field metamod-els, IEEE Trans. Evol. Comput. (Special Issue on EvolutionaryComputation in the Presence of Uncertainty) 10 (4) (2006) 421–439.

[32] Y.S. Ong, P.B. Nair, A.J. Keane, Evolutionary optimization ofcomputationally expensive problems via surrogate modeling, AIAA J.41 (4) (2003) 687–696.

[33] P.K.S. Nain, K. Deb, A multi-objective optimization procedure withsuccessive approximation methods, KANGAL Technical Report2005002, Kanpur Genetic Algorithms Laboratory, India, 2005.

[34] J. Knowles, E.J. Hughes, Multiobjective optimization on a budget of250 evaluations, in: Proceedings of Evolutionary Multi-CriterionOptimization Conference, 2005, LNCS 3410, pp. 176–190.

[35] J. Knowles, ParEGO: a hybrid algorithm with on-line landscapeapproximation for expensive multi-objective optimization problems,IEEE Trans. Evol. Comput. 10 (1) (2006) 50–66.

[36] Y.S. Chen, Compressible and incompressible flow computation with apressure-based method, AIAA89-0286, in: AIAA 28th AerospaceSciences Meeting, 1989.

[37] T.S. Wang, Y.S. Chen, A united Navier–Stokes flowfield andperformance analysis of liquid rocket engines, AIAA 90-2494, in:AIAA 26th Joint Propulsion Conference, 1990.

[38] Y.S. Chen, R.C. Farmer, CFD analysis of baffle flame stabilization,AIAA 91-1967, in: AIAA 27th Joint Propulsion Conference, 1991.

[39] R.H. Myers, D.C. Montgomery, Response Surface Methodology –Process and Product Optimization Using Designed Experiment,Wiley-Interscience, 1995.

[40] R.E. Steuer, Multiple Criteria Optimization: Theory, Computation,and Application, Wiley, New York, 1986.

[41] D.E. Goldberg, Genetic Algorithms in Search, Optimization andMachine Learning, Addison-Wesley, 1989.

[42] T. Goel, R. Vaidyanathan, R.T. Haftka, N.V. Queipo, W. Shyy, K.Tucker, Response surface approximation of Pareto optimal front inmulti-objective optimization, in: Proceedings of the 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference,Albany, NY, 2004, AIAA2004-4501.

[43] MATLAB�, The language of technical computing, Version 6.5Release 13. � 1984–2002, The MathWorks, Inc.

[44] P. Ramu, S. Missoum, R.T. Haftka, A convex hull approach for thereliability-based optimization of transient dynamic problems, in:Proceedings of the 10th AIAA/ISSMO Multidisciplinary Analysisand Optimization Conference, Albany, NY, 2004, AIAA 2004-4618.

[45] A.K. Jain, R.C. Dubes, Algorithms for Clustering Data, Prentice HallCollege, New Jersey, 1988.

[46] A. Owen, Orthogonal arrays for: computer experiments, integrationand visualization, Statist. Sin. 2 (2) (1992) 439–452.

[47] JMP�, The statistical discovery softwareTM, Version 5, Copyright �1989–2002, SAS Institute Inc., Cary, NC, USA.

[48] Y. Mack, T. Goel, W. Shyy, R.T. Haftka, Surrogate model basedoptimization framework: a case study in aerospace design, in: S.Yang, Y.S. Ong, Y. Jin (Eds.), Evolutionary Computation inDynamic and Uncertain Environments, accepted for publication.

[49] R. Vaidyanathan, T. Goel, W. Shyy, R.T. Haftka, N.V. Queipo, K.Tucker, Global sensitivity and trade-off analyses for multi-objectiveliquid rocket injector design, in: 40th AIAA/ASME/SAE/ASEE JointPropulsion Conference and Exhibit, Ft. Lauderdale, FL, 2004.

Documents

Response surface approximation of Pareto optimal front in multi-objective optimization