Fast sensitivity analysis methods for computationally ... · Sensitivity analysis is a powerful tool for understanding the behaviour of a numerical model. It allows quantiﬁcation

Geosci. Model Dev., 11, 3131–3146, 2018https://doi.org/10.5194/gmd-11-3131-2018© Author(s) 2018. This work is distributed underthe Creative Commons Attribution 4.0 License.

Fast sensitivity analysis methods for computationally expensivemodels with multi-dimensional outputEdmund Ryan1, Oliver Wild1, Apostolos Voulgarakis2, and Lindsay Lee3

1Lancaster Environment Centre, Lancaster University, Lancaster, UK2Department of Physics, Imperial College London, London, UK3School of Earth and Environment, University of Leeds, Leeds, UK

Correspondence: Edmund Ryan ([email protected])

Received: 30 October 2017 – Discussion started: 13 November 2017Revised: 30 May 2018 – Accepted: 6 June 2018 – Published: 3 August 2018

Abstract. Global sensitivity analysis (GSA) is a powerful ap-proach in identifying which inputs or parameters most affecta model’s output. This determines which inputs to includewhen performing model calibration or uncertainty analysis.GSA allows quantification of the sensitivity index (SI) of aparticular input – the percentage of the total variability inthe output attributed to the changes in that input – by aver-aging over the other inputs rather than fixing them at spe-cific values. Traditional methods of computing the SIs us-ing the Sobol and extended Fourier Amplitude SensitivityTest (eFAST) methods involve running a model thousandsof times, but this may not be feasible for computationallyexpensive Earth system models. GSA methods that use a sta-tistical emulator in place of the expensive model are popu-lar, as they require far fewer model runs. We performed aneight-input GSA, using the Sobol and eFAST methods, ontwo computationally expensive atmospheric chemical trans-port models using emulators that were trained with 80 runsof the models. We considered two methods to further reducethe computational cost of GSA: (1) a dimension reductionapproach and (2) an emulator-free approach. When the out-put of a model is multi-dimensional, it is common practiceto build a separate emulator for each dimension of the outputspace. Here, we used principal component analysis (PCA) toreduce the output dimension, built an emulator for each ofthe transformed outputs, and then computed SIs of the re-constructed output using the Sobol method. We consideredthe global distribution of the annual column mean lifetimeof atmospheric methane, which requires ∼ 2000 emulatorswithout PCA but only 5–40 emulators with PCA. We alsoapplied an emulator-free method using a generalised addi-

tive model (GAM) to estimate the SIs using only the trainingruns. Compared to the emulator-only methods, the emulator–PCA and GAM methods accurately estimated the SIs of the∼ 2000 methane lifetime outputs but were on average 24 and37 times faster, respectively.

1 Introduction

Sensitivity analysis is a powerful tool for understanding thebehaviour of a numerical model. It allows quantification ofthe sensitivity in the model outputs to changes in each of themodel inputs. If the inputs are fixed values such as modelparameters, then sensitivity analysis allows study of howthe uncertainty in the model outputs can be attributed tothe uncertainty in these inputs. Sensitivity analysis is impor-tant for a number of reasons: (i) to identify which parame-ters contribute the largest uncertainty to the model outputs,(ii) to prioritise estimation of model parameters from obser-vational data, (iii) to understand the potential of observationsas a model constraint and (iv) to diagnose differences in be-haviour between different models.

1.1 Different approaches for sensitivity analysis

By far, the most common types of sensitivity analysis arethose performed one at a time (OAT) and locally. OAT sensi-tivity analysis involves running a model a number of times,varying each input in turn, whilst fixing other inputs at theirnominal values. For example, Wild (2007) showed that thetropospheric ozone budget was highly sensitive to differencesin global NOx emissions from lightning. The observation-

Published by Copernicus Publications on behalf of the European Geosciences Union.

3132 E. Ryan et al.: Computationally expensive models with multi-dimensional output

based range of 3–8 TgN yr−1 in the magnitude of these emis-sions could result in a 10 % difference in predicted tropo-spheric ozone burden. OAT sensitivity analysis is used ina variety of research fields including environmental science(Bailis et al., 2005; Campbell et al., 2008; de Gee et al., 2008;Saltelli and Annoni, 2010), medicine (Coggan et al., 2005;Stites et al., 2007; Wu et al., 2013), economics (Ahtikoski etal., 2008) and physics (Hill et al., 2012). While the ease ofimplementing OAT sensitivity analysis is appealing, a majordrawback of this approach is that it assumes that the modelresponse to different inputs is independent, which in mostcases is unjustified (Saltelli and Annoni, 2010) and can re-sult in biased results (Carslaw et al., 2013).

Global sensitivity analysis (GSA) overcomes this OAT is-sue by quantifying the sensitivity of each input variable byaveraging over the other inputs rather than fixing them atnominal values. However, the number of sensitivity analysisstudies using this global method has been very small. Ferrettiet al. (2016) found that out of around 1.75 million researcharticles surveyed up to 2014, only 1 in 20 of studies mention-ing “sensitivity analysis” also use or refer to “global sensitiv-ity analysis”. A common type of GSA is the variance-basedmethod, which operates by apportioning the variance of themodel’s output into different sources of variation in the in-puts. More specifically, it quantifies the sensitivity of a partic-ular input – the percentage of the total variability in the out-put attributed to the changes in that input – by averaging overthe other inputs rather than fixing them at specific values. TheFourier Amplitude Sensitivity Test (FAST) was one of thefirst of these variance-based methods (Cukier et al., 1973).The classical FAST method uses spectral analysis to appor-tion the variance, after first exploring the input space usingsinusoidal functions of different frequencies for each inputfactor or dimension (Saltelli et al., 2012). Modified versionsof FAST include the extended FAST (eFAST) method whichimproves its computational efficiency (Saltelli et al., 1999)and the random-based-design (RBD) FAST method whichsamples from the input space more efficiently (Tarantola etal., 2006). Another widely used GSA method is the Sobolmethod (Homma and Saltelli, 1996; Saltelli, 2002; Sobol,1990), which has been found to outperform FAST (Saltelli,2002). Most applications of the Sobol and FAST methods in-volve a small number of input factors. However, Mara andTarantola (2008) carried out a 100-input sensitivity analysisusing the RBD version of FAST and a modified version of theSobol method and found that both methods gave estimates ofthe sensitivity indices (SIs) that were close to the known an-alytical solutions. A downside to the Sobol method is that alarge number of runs of the model typically need to be carriedout. For the model used in Mara and Tarantola (2008), 10 000runs were required for the Sobol method but only 1000 wereneeded for FAST.

1.2 Emulators and meta-models

If a model is computationally expensive, carrying out 1000simulations may not be feasible. A solution is to use a sur-rogate function for the model called a meta-model that mapsthe same set of inputs to the same set of outputs but is com-putationally much faster. Thus, much less time is requiredto perform GSA using the meta-model than using the slow-running model. A meta-model can be any function that mapsthe inputs of a model to its outputs, e.g. linear or quadraticfunctions, splines, neural networks. A neural network, for ex-ample, works well if there are discontinuities in the input–output mapping, but such a method can require thousandsof runs of the computationally expensive model to train it(particularly if the output is highly multi-dimensional) whichwill likely be too time-consuming. Here, we use a statisti-cal emulator because it requires far fewer training runs andit has two useful properties. First, an emulator is an inter-polating function which means that at inputs of the expen-sive model that are used to train the emulator, the resultingoutputs of the emulator must exactly match those of the ex-pensive model (Iooss and Lemaître, 2015). Secondly, for in-puts that the emulator is not trained at, a probability distribu-tion of the outputs that represents their uncertainty is given(O’Hagan, 2006). The vast majority of emulators are basedon Gaussian process (GP) theory due to its attractive prop-erties (Kennedy and O’Hagan, 2000; O’Hagan, 2006; Oak-ley and O’Hagan, 2004), which make GP emulators easy toimplement while providing accurate representations of thecomputationally expensive model (e.g. Chang et al., 2015;Gómez-Dans et al., 2016; Kennedy et al., 2008; Lee et al.,2013). A GP is a multivariate normal distribution appliedto a function rather than a set of variables. The original GPemulator in a Bayesian setting was developed by Currin etal. (1991) (for a basic overview, see also O’Hagan, 2006)and is mathematically equivalent to the kriging interpola-tion methods used in geostatistics (e.g. Cressie, 1990; Rip-ley, 2005). Kriging regression has been used as an emulatormethod since the 1990s (Koehler and Owen, 1996; Welchet al., 1992). More recently, there has been considerable in-terest in using this kriging emulator approach for practicalpurposes such as GSA or inverse modelling (Marrel et al.,2009; Roustant et al., 2012). Examples of its application canbe found in atmospheric modelling (Carslaw et al., 2013; Leeet al., 2013), medicine (Degroote et al., 2012) and electricalengineering (Pistone and Vicario, 2013).

For GSA studies involving multi-dimensional output, atraditional approach is to apply a separate GP emulator foreach dimension of the output space. However, if the outputconsists of many thousands of points on a spatial map or timeseries (Lee et al., 2013), then the need to use thousands ofemulators can impose substantial computational constraintseven using the FAST methods. A solution is to adopt a GSAmethod that does not rely on an emulator but is based ongeneralised additive modelling (Mara and Tarantola, 2008;

Geosci. Model Dev., 11, 3131–3146, 2018 www.geosci-model-dev.net/11/3131/2018/

E. Ryan et al.: Computationally expensive models with multi-dimensional output 3133

Strong et al., 2014, 2015b) or on a partial least squares ap-proach (Chang et al., 2015; Sobie, 2009). A separate gen-eralised additive model (GAM) can be built for each inputagainst the output of the expensive model, and the sensitivityof the output to changes in each input is then computed usingthese individual GAM models. Partial least squares (PLS) isan extension of the more traditional multivariate linear re-gression where the number of samples (i.e. model runs inthis context) can be small, and they may even be less that thenumber of inputs (Sobie, 2009).

An alternative way of reducing the computational con-straints is to use principal component analysis (PCA) to re-duce the dimensionality of the output. This means that werequire far fewer emulators to represent the outputs, reduc-ing the GSA calculations by a large margin, although there issome loss of detail. This emulator–PCA hybrid approach hasbeen successfully used in radiative transfer models (Gómez-Dans et al., 2016), a very simple chemical reaction model(Saltelli et al., 2012) and general circulation models (Sextonet al., 2012). While we hypothesise that both emulator-freeand PCA-based methods are suited to large-scale GSA prob-lems (e.g. those involving more than 20 input factors), a fo-cus of our work is to determine the accuracy of these methodsfor a smaller-scale GSA study.

1.3 Aims of this study

Recent research comparing different GSA methods basedon Gaussian process emulators has been limited in applica-tion to relatively simple models and low-dimensional out-put (Mara and Tarantola, 2008). Using two computation-ally expensive models of global atmospheric chemistry andtransport – namely the Frontier Research System for GlobalChange/University of California at Irvine (FRSGC/UCI) andGoddard Institute for Space Studies (GISS) models – wecompare the accuracy and efficiency of global sensitivityanalysis using emulators and emulator-free methods, and weinvestigate the benefits of using PCA to reduce the numberof emulators needed. We compare and contrast a number ofways of computing the first-order sensitivity indices for theexpensive atmospheric models: (i) the Sobol method usingan emulator, (ii) the extended FAST method using an emu-lator, (iii) generalised additive modelling, (iv) a partial leastsquares approach and (v) an emulator–PCA hybrid approach.Hereafter, we refer to (i) and (ii) as emulator-based GSAmethods and (iii) and (iv) as emulator-free GSA methods.

2 Materials and methods

2.1 Atmospheric chemistry models

Global atmospheric chemistry and transport models simu-late the composition of trace gases in the atmosphere (e.g.O3, CH4, CO, SOx) at a given spatial resolution (latitude× longitude × altitude). The evolution in atmospheric com-

position over time is controlled by a range of different dy-namical and chemical processes, our understanding of whichremains incomplete. Trace gases are emitted from anthro-pogenic sources (e.g. NO from traffic and industry) andfrom natural sources (e.g. isoprene from vegetation, NO fromlightning), they may undergo chemical transformation (e.g.formation of O3) and transport (e.g. convection or bound-ary layer mixing), and they may be removed through wet ordry deposition. Global sensitivity analysis is needed to under-stand the sensitivity of our simulations of atmospheric com-position and its evolution to assumptions about these govern-ing processes.

In this study, we performed GSA on two such atmosphericmodels. We used the FRSGC/UCI chemistry transport model(CTM) (Wild et al., 2004; Wild and Prather, 2000) andthe GISS general circulation model (GCM) (Schmidt et al.,2014; Shindell et al., 2006). We used results from 104 modelruns carried out with both of these models from a compar-ative GSA study (Wild et al., 2018). This involved vary-ing eight inputs or parameters over specified ranges using amaximin Latin hypercube design: global surface NOx emis-sions (30–50 TgN yr−1), global lightning NOx emissions (2–8 TgN yr−1), global isoprene emissions (200–800 TgC yr−1),dry deposition rates (model value ±80 %), wet depositionrates (model value ±80 %), humidity (model value ±50 %),cloud optical depth (model value× 0.1–10) and boundarylayer mixing (model value× 0.01–100). For this study, wefocus on a single model output, namely the global distribu-tion of tropospheric columns of mean methane (CH4) life-time at the annual timescale. The CH4 lifetime is an im-portant indicator of the amount of highly reactive hydroxylradical in the troposphere (Voulgarakis et al., 2013), and wechoose this output because of its contrasting behaviour inthe two models. The native spatial resolution of the modelsis 2.8◦× 2.8◦ for FRSGC and 2.5◦× 2.0◦ for GISS, but wecombine neighbouring grid points so that both models havea comparable resolution of 5–6◦, giving a total of 2048 gridpoints for FRSGC/UCI and 2160 grid points for GISS.

2.2 Global sensitivity analysis using the Sobol andextended FAST methods

For brevity and generality, we hereafter refer to each of theatmospheric chemical transport models as a simulator. Acommon way of conducting global sensitivity analysis foreach point in the output space of the simulator – where theoutput consists of, for example, a spatial map or a time series– is to compute the first-order sensitivity indices (SIs) usingvariance-based decomposition; this apportions the variancein simulator output (a scalar) to different sources of varia-tion in the different model inputs. Assuming the input vari-ables are independent of one another – which they are for thisstudy – the first-order SI, corresponding to the ith input vari-able (i = 1, 2, ..., p) and the j th point in the output space, is

www.geosci-model-dev.net/11/3131/2018/ Geosci. Model Dev., 11, 3131–3146, 2018


given by

Si,j =Var

[E(Y j |Xi

)]Var(Y j )

× 100, (1)

where Xi is the ith column of the n×p matrix X (i.e. a ma-trix with n rows and p columns) which stores the n samplesof p-dimensional inputs, and Y j is the j th column of then×m matrix which stores the corresponding n sets of m-dimensional outputs (Table 1). We multiply by 100 so thatthe SI is given as a percentage. The notation given by Var(·)and E(·) denotes the mathematical operations that computethe variance and expectation. The simplest way of computingSi,j is by brute force, but this is also the most computation-ally intensive (Saltelli et al., 2008).

2.2.1 The Sobol method

The Sobol method, developed in the 1990s, is much fasterthan brute force at computing the terms in Eq. (1), in part be-cause it requires fewer executions of the simulator (Hommaand Saltelli, 1996; Saltelli, 2002; Saltelli et al., 2008; Sobol,1990). The method operates by first generating a n×2p ma-trix (i.e. a matrix with n rows and 2p columns) of randomnumbers from a space-filling sampling design (e.g. a max-imin Latin hypercube design), where n is the number of setsof inputs and p is the number of input variables. The in-puts are on the normalised scale so that each element of ap-dimensional input lies between 0 and 1. Typical values forn are 1000–10 000. The matrix is split in half to form twonew matrices, A and B, each of size n×p. To compute theith SI (1 ≤ i ≤ p), we define two new matrices, Ci and Di,where Ci is formed by taking the ith column from A and theremaining columns from B, and Di is formed by taking theith column from B and the remaining columns from A. Wethen execute the simulator – denoted by f – at each set ofinputs given by the rows of matrices A, B, Ci and Di. Thisgives vectors YA = f (A), YB = f (B), YCi = f (Ci) andYDi = f (Di). Vectors YA and YCi are then substituted intoEq. (2):

ˆSi,j =Var

[E(Y j |Xi

)]Var(Y j )

× 100

=

YA ·YCi−

(1N

N∑j=1

Y(j)

A

)2

YA ·YA−

(1N

N∑j=1

Y(j)

A

)2 × 100, (2)

where YA ·YCi =

(1N

N∑j=1

Y(j)

A Y(j)

Ci

), and Y (j)A and Y (j)Ci are

the j th elements of YA and YCi (equivalent formula forYA ·YA). For all p input variables, the total number of sim-ulator runs is 12× n×p. Saltelli (2002) and Tarantola et

al. (2006) suggested using eight variants of Eq. (2), using dif-ferent combinations of YA, YB, YCi and YDi (Appendix A).Lilburne and Tarantola (2009) proposed using the averageof these eight SI estimates as they deemed this to be moreaccurate than a single estimate. We used this approach byLilburne and Tarantola (2009) for this study.

2.2.2 The extended FAST method

An alternative and even faster way of estimating the termsin Eq. (1) is to use the eFAST method, first developed bySaltelli et al. (1999) and widely used since (Carslaw et al.,2013; Koehler and Owen, 1996; Queipo et al., 2005; Saltelliet al., 2008; Vanuytrecht et al., 2014; Vu-Bac et al., 2015).A multi-dimensional Fourier transformation of the simulatorf allows a variance-based decomposition that samples theinput space along a curve defined by

xi(s)=Gi (sin(ωis)) , (3)

where x = (x1, ..., xp) refers to a general point in the inputspace that has been sampled, s ∈ R is a variable over therange (−∞,∞), Gi is the ith transformation function (Ap-pendix A), and ωi is the ith user-specified frequency corre-sponding to each input. Varying s allows a multi-dimensionalexploration of the input space due to the xis being simulta-neously varied. Depending on the simulator, we typically re-quire n= 1000–10 000 samples from the input space. Afterapplying the simulator f , the resulting scalar output – de-noted generally by y – produces different periodic functionsbased on differentωi . If the output y is sensitive to changes inthe ith input factor, the periodic function of y correspondingto frequency ωi will have a high amplitude.

More specifically, we express the model y = f (s)=f(x1 (s) ,x2 (s) , . . .,xp (s)

)as a Fourier series:

y = f (s)=

∞∑j=−∞

Aj cos(js)+Bj sin(js) . (4)

Using a domain of frequencies given by j ∈ Z =

{−∞, . . .,−1,0,1, . . .,∞}, the Fourier coefficients Aj andBj are defined by

Aj =1

2π

π∫−π

f (s)cos(js) .ds, Bj =1

2π

π∫−π

f (s)sin(js) .ds. (5)

With ωi stated in Eq. (3), the variance of model outputattributed to changes in the ith input variable for the j th pointin the output space (numerator of Eq. 1) is defined as

Var[E(Y j |Xi

)]=

∑q ∈Z0

A2qωi+B2

qωi, (6a)

where Z0 is the set of all integers except zero. The total vari-ance (denominator of Eq. 1) is

Var(Y j )=∑k∈Z0

A2k +B

2k . (6b)



Table 1. Summary of algebraic terms used in this study that are common to all of most of the statistical methods described in this study. Forbrevity, the terms that are specific to a particular method are not listed here.

Symbol Description

Si,j The first-order sensitivity index corresponding to the ith input variable (i = 1, 2, ..., p) and the j th point in theoutput space

n In general, n is the number of executions of the simulator required to compute the sensitivity indices. For thisstudy, n is the number of executions of the “emulator” required to compute the sensitivity indices since thesimulator is computationally too slow to run. For the Sobol and eFAST methods, n= 1000–10 000 (for thisstudy, we used n= 10 000 for Sobol and n= 5000 for eFAST). For the GAM and PLS methods, we believen < 100 is sufficient (for this study, we used n=N = 80)

p The number of input variables/the dimension of the input space

m The number of output variables/the dimension of the output space

N The number of executions of the simulator required to train an emulator (for this study, N = 80)

X Apart from Eq. (1), X refers to the N ×p matrix which stores the N sets of p-dimensional inputs that areused for two purposes: (i) in the calculations to train the emulators that are used to replace the simulator (seeSect. 2.3) and (ii) in the calculation of the sensitivity indices using the sensitivity analysis methods that do notrequire an emulator (namely GAM and PLS). For Eq. (1), X also refers to the n×p matrix to compute the SIsif the simulator is computationally cheap to run.

Xi A column vector represented by the ith column of matrix X (i = 1, 2, ..., p)

xi The row vector represented by the ith row of matrix X (i = 1, 2, ..., N)

Y The n×m matrix which stores the n sets of m-dimensional simulator outputs (corresponding to the n sets ofinputs stored in X) that are used as part of the calculation to compute the sensitivity indices

Y j The j th column of matrix Y (j = 1, 2, ..., m)

yi The simulator output after the simulator has been run at the p-dimensional input given by xi (i = 1, 2, ..., N)

Further details of eFAST are given in Saltelli et al. (1999).The differences between the original and the extended ver-sions of the FAST method are given in Appendix A.

2.3 Gaussian process emulators

When the simulator is computationally expensive to run –like the atmospheric chemical transport models used here –we substitute it with an emulator which is a surrogate of theexpensive simulator but much faster to run. If we are con-fident that the emulator is accurate, then we can computethe first-order SIs from the Sobol and eFAST methods usingthe outputs of the emulator rather than the simulator. Mathe-matically, an emulator is a statistical model that mimics theinput–output relationship of a simulator. As stated in the in-troduction, an emulator is an interpolating function at modeloutputs it is trained at and gives a probability distribution andother outputs (O’Hagan, 2006).

An emulator is trained using N sets of p-dimensional in-puts denoted by x1, x2, ..., xN andN sets of one-dimensionaloutputs from the simulator given by y1 = f (x1), y2 = f (x2),..., yN = f (xN ); f represents the simulator and for our studyN = 80 (see Sect. 2.6). The most common form of an emula-tor is a GP since it has attractive mathematical properties that

allow an analytical derivation of the mean and variance of theemulated output (given by f (x) for a general input x). A no-table exception is Goldstein and Rougier (2006), who useda non-GP emulator based on a Bayes linear approach. Moreformally, a GP is an extension of the multivariate Gaussiandistribution to infinitely many variables (Rasmussen, 2006).The multivariate Gaussian distribution is specified by a meanvector µ and covariance matrix6. A GP has a mean functionwhich is typically given by m(x)= E(f (x)) and covariancefunction given by c(x, x′) = cov(f (x), f (x′)), where x andx′ are two different p-dimensional inputs. For the latter, weused a Matern (5/2) function (Roustant et al., 2012), which isgiven by

c(x,x′

)= s2+

1+√

5

(∣∣x− x′∣∣θ

)+

53

(∣∣x− x′∣∣θ

)2

× exp

(−√

5

(∣∣x− x′∣∣θ

)), (7)

where s denotes the standard deviation and θ is the vectorof range parameters (sometimes called length scales). Theseemulator parameters are normally estimated using maximumlikelihood (see Bastos and O’Hagan, 2009, for details). GP



emulators for uncertainty quantification were originally de-veloped within a Bayesian framework (Currin et al., 1991;Kennedy and O’Hagan, 2000; O’Hagan, 2006; Oakley andO’Hagan, 2004).

Developed around the same time, the kriging interpolationmethods used in geostatistics are mathematically equivalentto the GP methods developed by Currin et al. (1991) (e.g.Cressie, 1990; Ripley, 2005). Kriging-based emulators havebeen used for 25 years (Koehler and Owen, 1996; Welch etal., 1992), with recent implementations including the DICE-Kriging R packages used for GSA and inverse modelling(Marrel et al., 2009; Roustant et al., 2012). Since the latterapproach is computationally faster, we adopted the DICE-Kriging version of the GP emulator for this study. For thestatistical theory behind both emulator versions and descrip-tions of related R packages, see Hankin (2005) and Roustantet al. (2012).

2.4 Emulator-free global sensitivity analysis

For GSA studies involving highly multi-dimensional output,the time to compute the SIs can be significantly reduced byemploying an emulator-free GSA approach. In this study, weconsider two such methods using (i) GAM and (ii) a PLS re-gression approach. For both the GAM and PLS methods, weused n=N simulator runs to compute the sensitivity indices(Table 1), and for our study these were the sameN = 80 runsthat were used to train the emulators described in Sect. 2.3.In the descriptions of these two sensitivity analysis methods(Sect. 2.4.1 and 2.4.2), we thus use X= [X1X2, . . .,Xp] andY to denote the matrices that store N sets of p-dimensionalinputs and m-dimensional outputs.

2.4.1 The generalised additive modelling method

A GAM is a generalised linear model where the predictorvariables are represented by smooth functions (Wood, 2017).The general form of a GAM is

Y j = g (X)+ ε (8a)g (X)= s (X1)+ s (X2)+ . . .+ s

(Xp), (8b)

where Xi is the ith column of input matrix X (i = 1, 2, ...,p); Y j is the j th column of output matrix Y (j = 1, 2, ...,m)since we construct a separate GAM for each point in the out-put space (i.e. for each latitude–longitude point in our case);s(.) is the smoothing function such as a cubic spline; and εis a zero-mean normally distributed error term with constantvariance. If we wish to include second-order terms in g (X),we would add s (X1,X2)+ s (X1,X3)+ . . .+ s

(Xp−1,Xp

)to the right-hand side of Eq. (8b). A GAM it is not an emu-lator as defined by O’Hagan (2006) because the fitted valuesof the GAM are not exactly equal to the outputs of the train-ing data (Simon N. Wood, personal communication, 23 May2017). It is still a meta-model and we could use it as a sur-rogate of the computationally expensive simulator in order

to perform variance-based sensitivity analysis using, for ex-ample, the Sobol or extended FAST method. However, wehave found that the number of runs of the simulator to train itin order for it to be an accurate surrogate for the simulator istoo many (i.e. too computationally burdensome). Instead, it ispossible to obtain accurate estimates of the first-order SIs byusing a GAM to estimate the components of Eq. (1) directly(Stanfill et al., 2015; Strong et al., 2014, 2015b). To com-pute the ith first-order SI (1≤ i ≤ p), we first recognise thattaking the expectation of Eq. (8a) leads to E

(Y j)= g(X).

The expression for E(Y j |Xi

)is thus the marginal distribu-

tion of E(Y j). We could fit the full model and then com-

pute this marginal distribution following Stanfill et al. (2015).However, an easier and quicker way is to fit a GAM to the(Xi , Y j ) “data” where Xi and Y j are defined above. Then,E(Y j |Xi

)consists of the fitted values of this reduced model

(Strong et al., 2015b). Thus, Var[E(Y j |Xi

)](numerator of

equation 1) is determined by computing the variance of the npoints from this fitted GAM model. In other words,

Var[E(Y j |Xi

)]= var

(s(x1,i

), s(x2,i

), . . ., s

(xn,i

)), (9)

where xk,i is the element from the kth row and ith columnof matrix X. Finally, the denominator term of Eq. (1) is com-puted by taking the variance of the n samples of the out-puts from the computationally expensive simulator that arestored in Y j .

2.4.2 The partial least squares method

The PLS method is the only one of the four GSA meth-ods considered here that is not variance-based (Chang et al.,2015). Multivariate linear regression (MLR) is a commonlyused tool to represent a set of outputs or response variables(Y) based on a set of inputs or predictor variables (X), whereX and Y are matrices (Table 1). MLR is only appropriate touse when the different inputs (columns in X) are indepen-dent and not excessive in number. In many situations, suchas GSA studies, there can be a large number of input vari-able and/or they could be highly correlated with each other(Sobie, 2009). PLS is an extension of MLR which is ableto deal with these more challenging multivariate modellingproblems (Wold et al., 2001). The main reason for choos-ing PLS over other applicable regression approaches is that ithas been shown to give similar estimates of the sensitivity in-dices to a variance-based GSA approach (Chang et al., 2015).Thus, for sensitivity analysis problems when the inputs arecorrelated, this PLS method could be considered an alterna-tive to the variance-based GAM method which assumes thatthe inputs are independent. Mathematically, PLS operates byprojecting X and Y into new spaces, determined by maximis-ing the covariance between the projections of X and Y (seeSect. S1 in the Supplement for details). PLS regression isthen performed where the regression coefficients representthe sensitivity indices (given as a percentage). When n > p,



it is standard to estimate the PLS regression coefficients us-ing the traditional multivariate linear regression. Thus, thep×m matrix of sensitivity indices (S) can be computed us-ing the following formula:

S =(

XTX)−1

XTY. (10)

2.5 Principal component analysis

As an alternative approach for speeding up the sensitivityanalysis calculations, we computed the SIs from the SobolGSA method using a hybrid approach involving PCA to re-duce the dimensionality of the output space, and then usedseparate Gaussian process emulators for each of the trans-formed outputs (Gómez-Dans et al., 2016; Saltelli et al.,2012; Sexton et al., 2012). After performing the emulatorruns, we then reconstruct the emulator output on the originaloutput space, from which we compute the sensitivity indices.

PCA transforms the outputs onto a projected space withmaximal variance. Mathematically, we obtain the matrix oftransformed outputs Y(PC) by

Y(PC)= YA∗, (11)

where Y is the N ×m matrix of training outputs from thesimulator (see Sect. 2.3), and A∗ is a matrix whose columnsare orthogonal to one another and whose ith column (A∗i ) ischosen such that var(YA∗i ) is maximised subject to the con-straint

(A∗i)TA∗i = 1. The vector A∗1 is called the first prin-

cipal component (PC1), and we define λ1 to be the principleeigenvalue of S = var (Y ) which is the largest variance ofthe outputs Y with respect to PC1. The second, third, fourthcolumns, etc. of A∗ are referred to as PC2, PC3, PC4, etc.with λ2, λ3, λ4, etc. representing the second, third, fourth,etc. largest variance of Y, respectively. PC1 contains themost information in the output, followed by PC2, then PC3,etc. The number of principal components required is com-monly determined by plotting the following points: (1, λ1),(2, λ1+ λ2), (3, λ1+ λ2+ λ3), ..., and identifying the pointwhere the line begins to flatten out. This is equivalent tochoosing a cutoff when most of the variance is explained.In this study, we included the first Npc principal componentssuch that 99 % of the variance is explained. The 99 % thresh-old was also necessary for this study to ensure that the recon-structed emulator output accurately approximated the simu-lator output for the validation runs (Fig. 2). While we foundthe 99 % threshold was necessary, other studies may find thata lower threshold (e.g. 95 %) is sufficient.

This technique of reducing the dimension of the out-put space from m=∼ 2000 spatially varying points to thefirst Npc principal components (e.g. Npc = 5 for the FRSGCmodel; see Sect. 2.6) means that the number of required em-ulator runs to compute the sensitivity indices from the Sobolmethod is reduced by a factor of m/Npc (=∼ 400 usingabove m and Npc values). However, after having generated

Figure 1. Flowchart for order of tasks to complete in order to per-form GSA on a computationally expensive model. The ranges on theinputs, on which its design is based, are determined by expert elici-tation. For approach 1, each dimension (dim.) of the output consistsof a different spatial or temporal point of the same variable (CH4lifetime for this study). For approach 2, a PC is a linear combina-tion of the different dimensions of the output, where n is chosensuch that the first n PCs explain 99 % of the variance of the output.

the Npc sets of output vectors for the Sobol method (Y (PC)A ,

Y(PC)B , Y (PC)

Ci , Y (PC)Di ; see Sect. 2.2), we need to reconstruct

the m sets of output vectors which are required to computethe sensitivity indices for each of the m points in the outputspace. To do this, we first set the elements of the (Npc+1)th,(Npc+ 2)th, ..., m columns of the matrix A∗ (Eq. 11) to zeroand call this new matrix A∗sample. We also form a n×mmatrix

Y(PC)sample whose first Npc columns are vectors storing the em-

ulator outputs corresponding to the first Npc principal com-ponents, while the elements of the remaining columns areset to zero. Recall that Y(PC)

sample is different from Y(PC) wherethe latter has N rows (80 for this study) which correspondto the number of simulator runs required to train the emula-tors, whereas the number of samples n (n= 10 000 for thisstudy) refers to the number of emulator runs needed to esti-mate the sensitivity indices. The n×m matrix Ysample of thereconstructed m-dimensional outputs is computed using

Ysample = Y(PC)sample

(A∗sample

)T. (12)

We use this formula to compute the YA, YB, YCi and YDivectors from Sect. 2.2 and the resulting sensitivity indicesusing Eq. (2) from the Sobol method (Sect. 2.2).



2.6 Experimental setup

The sequence of tasks to complete when performing globalsensitivity analysis is shown schematically in Fig. 1. Thechoice of inputs (e.g. parameters) to include in the sensi-tivity analysis will depend upon which have the greatest ef-fects, based on expert knowledge of the model and field ofstudy. Expert judgement is also needed to define the rangesof these inputs. A space-filling design such as maximin Latinhypercube sampling or sliced Latin hypercube sampling (Baet al., 2015) is required in order to sample from the inputspace with the minimum sufficient number of model runs.We used n= 10 000 for the Sobol method and n= 5000 forthe eFAST method, but n=N = 80 for the GAM and PLSmethods. The third stage is to run the model at the set of inputpoints specified by the space-filling sampling design.

If we are employing an emulator, the next stage is to buildthe emulator using the training runs. The number of trainingruns (N) is determined by N = 10×p, where p is the num-ber of input variables (Loeppky et al., 2009). We also needto perform runs of the computationally expensive simulatorto validate the emulators. For this study, we ran the simula-tors with an additional set of inputs for validation. Compar-ing the emulator outputs with the simulator outputs using thevalidation inputs is usually sufficient, but more sophisticateddiagnostics can also be carried out if needed (Bastos andO’Hagan, 2009). If employing the emulator-free approach,validation is also needed because we are using a statisticalmodel to infer the SIs. Such a validation is not a central partof our results but is included in the Supplement (Fig. S2). Forthe emulator–PCA hybrid approach (Fig. 1), we found thatthe first 5 (for FRSGC) and 40 (for GISS) principal compo-nents were required to account for 99 % of the variance. Thismeans that only 5–40 emulators are required to generate aglobal map in place of ∼ 2000 needed if each grid point isemulated separately, thus providing large computational sav-ings.

The final stage is to compute the first-order SIs for all theinputs; these quantify the sensitivity of the output to changesin each input. The SIs are also known as the main effects.The eFAST, Sobol and GAM approaches can also be usedto compute the total effects, defined as the sum of the sen-sitivities of the output to changes in input i on its own andinteracting with other inputs. For this study, we do not con-sider total effects as the sum of the main effects was close to100 % in each case.

3 Results

3.1 Validation of the emulators

Since the emulators we employed are based on a scalar out-put, we built a separate emulator for each of the ∼ 2000model grid points to represent the spatial distribution of the

Figure 2. Annual column mean CH4 lifetime calculated by theFRSGC and GISS chemistry models from each of 24 validation runs(x axis) versus that predicted by the emulator (y axis). In each plot,the R2 and median absolute difference (MAD) are given as metricsfor the accuracy of the emulator predictions. Each validation runcontains ∼ 2000 different output values, corresponding to differentlatitude–longitude grid squares.

CH4 lifetimes. At the 24 sets of inputs set aside for emula-tor validation, the predicted outputs from the emulators com-pared extremely well with the corresponding outputs fromboth chemistry models (Fig. 2a, b, R2

= 0.9996–0.9999, me-dian absolute difference of 0.1–0.18 years). When PCA isused to reduce the output dimension from ∼ 2000 to 5–40(depending on the chemistry model), the accuracy of the pre-dicted outputs was not as good (Fig. 2c, d, R2

= 0.9759–0.9991, median absolute difference of 0.94–3.44 years) butwas still sufficient for this study.

3.2 Comparison of sensitivity indices

As expected, the two emulator-based global sensitivity anal-ysis (GSA) approaches (eFAST and Sobol) produced almostidentical global maps of first-order SIs (%) of CH4 lifetime;see Figs. 3 and 4. The statistics (mean, 95th percentile and99th percentile) of the differences in SIs between the twoGSA methods over all eight inputs at 2000 output points forthe FRSGC and GISS models are shown in Fig. 5 (M1 ver-sus M2).

Our results show that the GAM emulator-free GSAmethod produces very similar estimates of the SIs to theemulator-based methods (Figs. 3, 4; row a vs. c for Sobolversus GAM). The 95th and 99th percentiles of differencesof the emulator-based methods (e.g. Sobol) versus GAM are5 and 9 % for FRSGC, and 7 and 10 % for GISS (Fig. 5, M1versus M3). For both models, the PLS non-emulator-basedmethod produced SIs that were significantly different fromthose using the eFAST and Sobol methods (Figs. 3, 4; rowa vs. d for Sobol vs. PLS). For FRSGC, the mean and 95thpercentile of the differences in SIs for the Sobol versus PLS



Figure 3. The sensitivity indices (percentage of the total variance in a given output) for the four dominant inputs, with the output given asthe annual column mean CH4 lifetime from the FRSGC chemistry transport model. The rows show the results from five different methodsfor performing sensitivity analysis (SA), whose formulae for computing the SIs are given by Eqs. (1, 2) and Sect. 2.3 (Sobol method andemulator), Eqs. (1, 6a–b), Sect. 2.3 (eFAST method and emulator), Eqs. (1, 9) (GAM method), Eq. (10) (PLS method), Eqs. (1, 2), Sect. 2.3and 2.5 (Sobol method, emulator and PCA).

Figure 4. The sensitivity indices (percentage of the total variance in a given output) for the four dominant inputs, with the output given asthe annual column mean CH4 lifetime from the GISS chemistry model. See caption for Fig. 3 for further details about the five methods used.



Figure 5. Statistics (mean, 95th percentile and 99th percentile) of the distribution of differences in sensitivity indices (SIs) between pairs ofmethods. For each comparison, the ∼ 16 000 pairs of SIs are made up of ∼ 2000 pairs of SIs for each of the eight inputs.

methods are around 21 and 31 %, while for GISS the cor-responding values are around 14 and 23 % (Fig. 5, M1 ver-sus M4). Thus, our results indicate that the PLS method isnot suitable for use as an emulator-free approach to estimat-ing the SIs.

The global map of SIs using the emulator–PCA hybrid ap-proach compared well to those from the emulator-only ap-proach (Figs. 3, 4; row a vs. e). The 95th and 99th percentilesof differences between the two approaches were 6 and 10 %,respectively, for FRSGC (Fig. 5a, M1 versus M5) and 3and 5 %, respectively, for GISS (Fig. 5b, M1 versus M5).These are both higher than the corresponding values forthe emulator-only methods (Fig. 5, M1 versus M2; < 2 and< 3 %, respectively). These higher values for the emulator–PCA hybrid approach are also reflected in the poorer esti-mates of the validation outputs using this approach versusthe emulator-only approach (Fig. 2). Such poorer estimatesare expected because the PCA-transformed outputs only ex-plain 99 % of the variance of the untransformed outputs usedin the emulator-only approach.

4 Discussion

4.1 Comparison of sensitivity indices

Our results align with the consensus that the eFAST methodor other modified versions of the FAST method (e.g. RBD-FAST) produce very similar SIs to the Sobol method. Math-ematically, the two methods are equivalent (Saltelli et al.,2012) and when the analytical (true) values of the SIs canbe computed, both methods are able to accurately estimatethese values (Iooss and Lemaître, 2015; Mara and Taran-tola, 2008). However, many studies have noted that the Sobolmethod requires more simulator (or emulator) runs to com-pute the SIs. Saltelli et al. (2012) state that 2

k×100 (%) more

model runs are required for the Sobol method compared toeFAST, where k is the number of input factors (e.g. if k = 8,then 25 % more runs are needed for Sobol). Mara and Taran-tola (2008) found that the Sobol method required ∼ 10 000runs of their model to achieve the same level of aggregated

absolute error to that of FAST, which only needed 1000 runs.This is comparable to our analysis where the Sobol methodrequired 18 000 runs of the emulator but only 1000 runs wereneeded for the eFAST method.

Given recent interest in applying GAMs to perform GSA(Strong et al., 2015a, b, 2014), only Stanfill et al. (2015) havecompared how they perform against other variance-based ap-proaches. The authors found that first-order SIs estimatedfrom the original FAST method were very close to the truevalues using 600 executions of the model, whereas the GAMapproach only required 90–150 model runs. This is roughlyconsistent with our results, as we estimated the SIs using 80runs of the chemistry models for GAM and 1000 runs of theemulator for the eFAST method.

There are a limited number of studies comparing the accu-racy of the SIs of the GAM method amongst different mod-els, as in our study. Stanfill et al. (2015) found that the GAMmethod was accurate at estimating SIs based on a simplemodel (three to four parameters) as well as a more complexone (10 parameters). However, if more models of varyingcomplexity and type (e.g. process versus empirical) were toapply the GAM approach, we expect that while GAM wouldwork well for some models, for others the resulting SIs maybe substantially different from those produced using the moretraditional Sobol or eFAST methods. Saltelli et al. (1993)suggests that the performance of a GSA method can be modeldependent, especially when the model is linear versus non-linear or monotonic versus non-monotonic, or if transforma-tions are applied on the output (e.g. logarithms) or not. Thisis particularly true for GSA methods based on correlationor regression coefficients (Saltelli et al., 1999), which mightexplain why the SIs calculated from the PLS method in ouranalysis also disagreed with those of the eFAST/Sobol meth-ods for the FRSGC versus GISS models. Not all GSA meth-ods are model dependent; for example, the eFAST method isnot (Saltelli et al., 1999).

4.2 Principal component analysis

For both chemistry models, using PCA to significantly re-duce the number of emulators needed resulted in SIs very



similar to those calculated using an emulator-only approach.For the GISS model, this was encouraging given that thespread of points and their bias in the emulator versus sim-ulator scatter plot were noticeably larger than those of theFRSGC model (Fig. 2c, d). If we had increased the num-ber of principle components so that 99.9 % of the variance inthe output was captured rather than 99 %, following Verrelstet al. (2016), then we would expect less bias in the valida-tion plot for GISS. However, the poor validation plots didnot translate into poorly estimated SIs for the emulator–PCAapproach. On the contrary, the estimated SIs for GISS areconsistent with the estimated SIs using either emulator-onlyapproach (Fig. 5).

The use of PCA in variance-based global sensitivity anal-ysis studies is relatively new but has great potential for appli-cation in other settings. De Lozzo and Marrel (2017) used anatmospheric gas dispersion model to simulate the evolutionand spatial distribution of a radioactive gas into the atmo-sphere following a chemical leak. The authors used principalcomponent analysis to reduce the dimension of the spatio-temporal output map of gas concentrations to speed up thecomputation of the Sobol sensitivity indices for each of the∼ 19 000 points in the output space. This emulator–PCA hy-brid approach was also used to estimate the Sobol sensitivityindices corresponding to a flood forecasting model that sim-ulates the water level of a river at 14 different points alongits length (Roy et al., 2017). Using a crop model to simulatea variable related to nitrogen content of a crop over a grow-ing season of 170 days, Lamboni et al. (2011) using PCAto reduce the dimension of the output space. However, un-like other comparable studies, the computed sensitivity in-dices corresponded to the principal components, i.e. to a lin-ear combination of the 170 output values. This is permissi-ble if the principal components can be interpreted in somephysical sense. For Lamboni et al. (2011), the first PC ap-proximately corresponded to mean nitrogen content over thewhole growing season, while the second PC was the differ-ence in nitrogen content between the first and second halvesof the growing season.

4.3 Scientific context of this study

Our work extends the work of Wild et al. (2018) who usedthe same training inputs and the same atmospheric chemicaltransport models (FRSGC and GISS) but different outputs.Instead of using highly multi-dimensional output of tropo-spheric methane lifetime values at different spatial locations,Wild et al. (2018) used a one-dimensional output of globaltropospheric methane lifetime. Using the eFAST method, theauthors found that global methane lifetime was most sensi-tive to change in the humidity input for the FRSGC model,while for the GISS model the surface NOx and the lightningNOx inputs were most important for predicting methane life-time at the global scale, followed by the isoprene, the bound-ary layer mixing and the humidity inputs (Wild et al., 2018).

As expected, our results indicated that these same inputs ex-plained most of the variance in the outputs for the differentspatial locations. However, while the humidity SI for GISSwas very low at the global scale (SI of 5 %), out study foundthat the SIs for humidity were very high (50–60 %) for thehigher-latitude regions (Fig. 4).

4.4 Implications for large-scale sensitivity analysisstudies

GSA studies for computationally expensive models involv-ing a small number of inputs (e.g. < 10) are useful andstraightforward to implement (Lee et al., 2012). However,the inferences made are limited due to the large number ofparameters on which these models depend and the numberof processes that they simulate. Hence, interest is growingin carrying out large-scale GSA studies involving a highnumber of inputs to improve understanding of an individ-ual model (e.g. Lee et al., 2013) or to diagnose differencesbetween models (Wild et al., 2018). For GSA studies whenthe number of inputs is small, our study has demonstratedthat the GAM approach is a good candidate for carrying outemulator-free GSA since it calculates very similar SIs with-out the computational demands of emulation. A caveat is thatthe performance of GAM may depend on the behaviour ofthe model; although we have found it is a good GSA methodfor our models (FRSGC and GISS) and output (CH4 life-times), its suitability may not be as good in all situations.

5 Conclusions

GSA is a powerful tool for understanding model behaviour,for diagnosing differences between models and for determin-ing which parameters to choose for model calibration. In thisstudy, we compared different methods for computing first-order sensitivity indices for computationally expensive mod-els based on modelled spatial distributions of CH4 lifetimes.We have demonstrated that the more established emulator-based methods (eFAST and Sobol) can be used to efficientlyderive meaningful sensitivity indices for multi-dimensionaloutput from atmospheric chemistry transport models. Wehave shown that an emulator-free method based on a GAMand an emulator–PCA hybrid method produce first-ordersensitivity indices that are consistent with the emulator-onlymethods. For a reasonably smooth system with few parame-ters, as investigated here, the GAM and PCA methods areviable and effective options for GSA, and are robust overmodels that exhibit distinctly different responses. Moreover,the computational benefit of these alternative methods isapparent, with the GAM approach allowing calculation ofvariance-based sensitivity indices 22–56 times faster (or 37times faster on average) compared to the eFAST or Sobolmethods. Using the Sobol method, the emulator–PCA hybridapproach is 19–28 times faster (or 24 times faster on aver-



age) at computing the sensitivity indices compared to usingan emulator-only approach depending on which chemistrymodel is used. Finally, we have provided guidance on how toimplement these methods in a reproducible way.

Code and data availability. The R code to carry out global sensi-tivity analysis using the methods described in this paper is avail-able in Sects. S2–S7 of the Supplement. This R code as well asthe R code used to validate the emulators can also be found viahttps://doi.org/10.5281/zenodo.1038667 (Ryan, 2017).

The inputs and outputs of the FRSGC chemistry model thatwere used to train the emulators in this paper can be found viahttps://doi.org/10.5281/zenodo.1038670 (Ryan and Wild, 2017).


https://doi.org/10.5281/zenodo.1038667



Appendix A: Further details of the Sobol and eFASTglobal sensitivity analysis methods

For the Sobol method, Saltelli (2002) and Tarantola etal. (2006) suggest using eight variants of Eq. (2), using dif-ferent combinations of yA, yB, yCi and yDi :

SiI=

YA ·YCi−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

YA ·YA−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

SiI I=

YB ·YDi−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

YB ·YB−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

SiIII=

YA ·YCi−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

YB ·YB−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

SiIV=

YA ·YCi−

(1N

N∑j=1

Y(j)

Ci

)(1N

N∑j=1

Y(j)

Di

)

YCi ·YCi−

(1N

N∑j=1

Y(j)

Ci

)(1N

N∑j=1

Y(j)

Di

)

SiV=

YA ·YCi−

(1N

N∑j=1

Y(j)

Ci

)(1N

N∑j=1

Y(j)

Di

)

YDi ·YDi−

(1N

N∑j=1

Y(j)

Ci

)(1N

N∑j=1

Y(j)

Di

)

SiVI=

YB ·YDi−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

YA ·YA−

(1N

N∑j=1

Y(j)

A

)(1N

N∑j=1

Y(j)B

)

SiVII=

YB ·YDi−

(1N

N∑j=1

Y(j)

Ci

)(1N

N∑j=1

Y(j)

Di

)

YCi ·YCi−

(1N

N∑j=1

Y(j)

Ci

)(1N

N∑j=1

Y(j)

Di

)

SiVIII=

YB ·YDi−

(1N

N∑j=1

Y(j)Ci

)(1N

N∑j=1

Y(j)Di

)

YDi ·YDi−

(1N

N∑j=1

Y(j)Ci

)(1N

N∑j=1

Y(j)Di

) .

Thus, the ith first-order Sobol SI estimate is

Si =18

(Si

I+ Si

II+ Si

III+ Si

IV+ Si

V+ Si

VI+ Si

VII+ Si

VIII).

The main difference between classical FAST (Cukier et al.,1973) and extended FAST (Saltelli et al., 1999) when com-puting first-order SIs is the choice of transformation functionGi :

classical FAST :Gi(z)= xievsz,(xi,vs are user-specified) (A1a)

extended FAST :Gi (z)=12+

1π

arcsin(z). (A1b)

Using Eq. (A1b), Eq. (3) now becomes a straight-lineequation:

xi(s)=12+

1πωis. (A2)



Supplement. The supplement related to this article is availableonline at: https://doi.org/10.5194/gmd-11-3131-2018-supplement.

Author contributions. ER and OW designed the study. ER con-ducted the analysis and wrote the manuscript, and OW gave feed-back during the analysis and writing phases. OW, FO and AW pro-vided output from the global atmospheric model runs needed tocarry out the analysis. LL advised on statistical aspects of the anal-ysis. All coauthors gave feedback on drafts of the manuscript.

Competing interests. The authors declare that they have no conflictof interest.

Acknowledgements. This work was supported by the NaturalEnvironment Research Council (grant number NE/N003411/1).

Edited by: Andrea StenkeReviewed by: three anonymous referees

References

Ahtikoski, A., Heikkilä, J., Alenius, V., and Siren, M.: Economic vi-ability of utilizing biomass energy from young stands – the caseof Finland, Biomass Bioenerg., 32, 988–996, 2008.

Ba, S., Myers, W. R., and Brenneman, W. A.: Optimal sliced Latinhypercube designs, Technometrics, 57, 479–487, 2015.

Bailis, R., Ezzati, M., and Kammen, D. M.: Mortality and green-house gas impacts of biomass and petroleum energy futures inAfrica, Science, 308, 98–103, 2005.

Bastos, L. S. and O’Hagan, A.: Diagnostics for Gaussian processemulators, Technometrics, 51, 425–438, 2009.

Campbell, J. E., Carmichael, G. R., Chai, T., Mena-Carrasco, M.,Tang, Y., Blake, D., Blake, N., Vay, S. A., Collatz, G. J., andBaker, I.: Photosynthetic control of atmospheric carbonyl sulfideduring the growing season, Science, 322, 1085–1088, 2008.

Carslaw, K., Lee, L., Reddington, C., Pringle, K., Rap, A., Forster,P., Mann, G., Spracklen, D., Woodhouse, M., and Regayre, L.:Large contribution of natural aerosols to uncertainty in indirectforcing, Nature, 503, 67–71, 2013.

Chang, E. T., Strong, M., and Clayton, R. H.: Bayesiansensitivity analysis of a cardiac cell model using aGaussian process emulator, PloS one, 10, e0130252,https://doi.org/10.1371/journal.pone.0137004, 2015.

Coggan, J. S., Bartol, T. M., Esquenazi, E., Stiles, J. R., Lamont, S.,Martone, M. E., Berg, D. K., Ellisman, M. H., and Sejnowski, T.J.: Evidence for ectopic neurotransmission at a neuronal synapse,Science, 309, 446–451, 2005.

Cressie, N.: The origins of kriging, Math. Geol., 22, 239–252, 1990.Cukier, R., Fortuin, C., Shuler, K. E., Petschek, A., and Schaibly,

J.: Study of the sensitivity of coupled reaction systems to uncer-tainties in rate coefficients. I Theory, The J. Chem. Phys., 59,3873–3878, 1973.

Currin, C., Mitchell, T., Morris, M., and Ylvisaker, D.: Bayesianprediction of deterministic functions, with applications to the de-

sign and analysis of computer experiments, J. Am. Stat. Assoc.,86, 953–963, 1991.

de Gee, M., Lof, M. E., and Hemerik, L.: The effect of chemicalinformation on the spatial distribution of fruit flies: II parame-terization, calibration, and sensitivity, B. Math. Biol., 70, 1850–1868, 2008.

Degroote, J., Couckuyt, I., Vierendeels, J., Segers, P., and Dhaene,T.: Inverse modelling of an aneurysm’s stiffness using surrogate-based optimization and fluid-structure interaction simulations,Struct. Multidis. Optim., 46, 457–469, 2012.

De Lozzo, M. and Marrel, A.: Sensitivity analysis with dependenceand variance-based measures for spatio-temporal numerical sim-ulators, Stoch. Environ. Res. Risk Assess., 31, 1437–1453, 2017.

Ferretti, F., Saltelli, A., and Tarantola, S.: Trends in sensitivity anal-ysis practice in the last decade, Sci. Total Environ., 568, 666–670, https://doi.org/10.1016/j.scitotenv.2016.02.133, 2016.

Goldstein, M. and Rougier, J.: Bayes linear calibrated prediction forcomplex systems, J. Am. Stat. Assoc., 101, 1132–1143, 2006.

Gómez-Dans, J. L., Lewis, P. E., and Disney, M.: Efficient Em-ulation of Radiative Transfer Codes Using Gaussian Processesand Application to Land Surface Parameter Inferences, RemoteSens., 8, 1–32, 2016.

Hankin, R. K.: Introducing BACCO, an R package for Bayesiananalysis of computer code output, J. Stat. Softw., 14, 1–21, 2005.

Hill, T. C., Ryan, E., and Williams, M.: The use of CO2 flux timeseries for parameter and carbon stock estimation in carbon cycleresearch, Global Change Biol., 18, 179–193, 2012.

Homma, T. and Saltelli, A.: Importance measures in global sensi-tivity analysis of nonlinear models, Reliab. Eng. Syst. Safe., 52,1–17, 1996.

Iooss, B. and Lemaître, P.: A review on global sensi-tivity analysis methods, in: Uncertainty Management inSimulation-Optimization of Complex Systems, Operations Re-search/Computer Science Interfaces Series, Vol 59. Springer,Boston, MA, 2015.

Kennedy, M., Anderson, C., O’Hagan, A., Lomas, M., Woodward,I., Gosling, J. P., and Heinemeyer, A.: Quantifying uncertainty inthe biospheric carbon flux for England and Wales, J. Royal Stat.Soc. A , 171, 109–135, 2008.

Kennedy, M. C. and O’Hagan, A.: Predicting the output from acomplex computer code when fast approximations are available,Biometrika, 87, 1–13, 2000.

Koehler, J. and Owen, A.: 9 Computer experiments, Handbook ofStatistics, 13, 261–308, 1996.

Lamboni, M., Monod, H., and Makowski, D.: Multivariate sensi-tivity analysis to measure global contribution of input factors indynamic models, Reliab. Eng. Syst. Safe., 96, 450–459, 2011.

Lee, L. A., Carslaw, K. S., Pringle, K. J., and Mann, G. W.:Mapping the uncertainty in global CCN using emulation, At-mos. Chem. Phys., 12, 9739–9751, https://doi.org/10.5194/acp-12-9739-2012, 2012.

Lee, L. A., Pringle, K. J., Reddington, C. L., Mann, G. W., Stier,P., Spracklen, D. V., Pierce, J. R., and Carslaw, K. S.: The mag-nitude and causes of uncertainty in global model simulations ofcloud condensation nuclei, Atmos. Chem. Phys., 13, 8879–8914,https://doi.org/10.5194/acp-13-8879-2013, 2013.

Lilburne, L. and Tarantola, S.: Sensitivity analysis of spatial models,Int. J. Geogr. Inform. Sci., 23, 151–168, 2009.


https://doi.org/10.5194/gmd-11-3131-2018-supplement

https://doi.org/10.1371/journal.pone.0137004

https://doi.org/10.1016/j.scitotenv.2016.02.133

https://doi.org/10.5194/acp-12-9739-2012

https://doi.org/10.5194/acp-12-9739-2012

https://doi.org/10.5194/acp-13-8879-2013


Loeppky, J. L., Sacks, J., and Welch, W. J.: Choosing the samplesize of a computer experiment: A practical guide, Technometrics,51, 366–376, 2009.

Mara, T. A. and Tarantola, S.: Application of global sensitivity anal-ysis of model output to building thermal simulations, BuildingSimulation, 1, 290–302, 2008.

Marrel, A., Iooss, B., Laurent, B., and Roustant, O.: Calculations ofsobol indices for the gaussian process metamodel, Reliab. Eng.Syst. Safe., 94, 742–751, 2009.

Oakley, J. E. and O’Hagan, A.: Probabilistic sensitivity analysis ofcomplex models: a Bayesian approach, J. Royal Stat. Soc. B, 66,751–769, 2004.

O’Hagan, A.: Bayesian analysis of computer code outputs: a tuto-rial, Reliab. Eng. Syst. Safe., 91, 1290–1300, 2006.

Pistone, G. and Vicario, G.: Kriging prediction from a circular grid:application to wafer diffusion, Appl. Stoch. Models Business In-dustry, 29, 350–361, 2013.

Queipo, N. V., Haftka, R. T., Shyy, W., Goel, T., Vaidyanathan, R.,and Tucker, P. K.: Surrogate-based analysis and optimization,Prog. Aerosp. Sci., 41, 1–28, 2005.

Rasmussen, C. E. and Williams, C. K. I.: Gaussian Processes forMachine Learning, 2006, the MIT Press, ISBN 026218253X,2006.

Ripley, B. D.: Spatial statistics, John Wiley & Sons, Hoboken, NewJersey, 2005.

Roustant, O., Ginsbourger, D., and Deville, Y.: DiceKriging,DiceOptim: Two R packages for the analysis of computer exper-iments by kriging-based metamodeling and optimization, avail-able at: https://hal.archives-ouvertes.fr/hal-00495766/document(last access: 15 June 2016), 2012.

Roy, P. T., El Moçayd, N., Ricci, S., Jouhaud, J.-C., Goutal, N.,De Lozzo, M., and Rochoux, M. C.: Comparison of PolynomialChaos and Gaussian Process surrogates for uncertainty quantifi-cation and correlation estimation of spatially distributed open-channel steady flows, Stoch. Environ. Res. Risk Assess., 2017,1–19, 2017.

Ryan, E.: Fast sensitivity analysis methods for computa-tionally expensive models with multi-dimensional output,https://doi.org/10.5281/zenodo.1038667, 2017.

Ryan, E. and Wild, O.: Data for the GSA methods paper by Ryan etal., https://doi.org/10.5281/zenodo.1038670, 2017.

Saltelli, A.: Making best use of model evaluations to compute sen-sitivity indices, Comput. Phys. Commun., 145, 280–297, 2002.

Saltelli, A., Andres, T., and Homma, T.: Sensitivity analysis ofmodel output: an investigation of new techniques, Comput. Stat.Data Anal., 15, 211–238, 1993.

Saltelli, A. and Annoni, P.: How to avoid a perfunctory sensitivityanalysis, Environ. Modell. Softw., 25, 1508–1517, 2010.

Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J.,Gatelli, D., Saisana, M., and Tarantola, S.: Global sensitivityanalysis: the primer, John Wiley & Sons, 2008.

Saltelli, A., Ratto, M., Tarantola, S., and Campolongo, F.: Update1 of: Sensitivity analysis for chemical models, Chem. Rev., 112,PR1–PR21, 2012.

Saltelli, A., Tarantola, S., and Chan, K.-S.: A quantitative model-independent method for global sensitivity analysis of model out-put, Technometrics, 41, 39–56, 1999.

Schmidt, G. A., Kelley, M., Nazarenko, L., Ruedy, R., Russell, G.L., Aleinov, I., Bauer, M., Bauer, S. E., Bhat, M. K., and Bleck,

R.: Configuration and assessment of the GISS ModelE2 contri-butions to the CMIP5 archive, J. Adv. Model. Earth Syst., 6, 141–184, 2014.

Sexton, D. M., Murphy, J. M., Collins, M., and Webb, M. J.: Multi-variate probabilistic projections using imperfect climate modelspart I: outline of methodology, Clim. Dynam., 38, 2513–2542,2012.

Shindell, D. T., Faluvegi, G., Unger, N., Aguilar, E., Schmidt,G. A., Koch, D. M., Bauer, S. E., and Miller, R. L.: Simula-tions of preindustrial, present-day, and 2100 conditions in theNASA GISS composition and climate model G-PUCCINI, At-mos. Chem. Phys., 6, 4427–4459, https://doi.org/10.5194/acp-6-4427-2006, 2006.

Sobie, E. A.: Parameter sensitivity analysis in electrophysiologicalmodels using multivariable regression, Biophys. J., 96, 1264–1274, 2009.

Sobol, I. Y. M.: On sensitivity estimation for nonlinear mathemati-cal models, Matemat. Modeliro., 2, 112–118, 1990.

Stanfill, B., Mielenz, H., Clifford, D., and Thorburn, P.: Simple ap-proach to emulating complex computer models for global sensi-tivity analysis, Environ. Modell. Softw., 74, 140–155, 2015.

Stites, E. C., Trampont, P. C., Ma, Z., and Ravichandran, K. S.:Network analysis of oncogenic Ras activation in cancer, Science,318, 463–467, 2007.

Strong, M., Oakley, J. E., and Brennan, A.: An efficient method forcomputing the Expected Value of Sample Information, A non-parametric regression approach, ScHARR working paper, 2015a.

Strong, M., Oakley, J. E., and Brennan, A.: Estimating multiparam-eter partial expected value of perfect information from a prob-abilistic sensitivity analysis sample a nonparametric regressionapproach, Med. Decis. Mak., 34, 311–326, 2014.

Strong, M., Oakley, J. E., Brennan, A., and Breeze, P.: Estimatingthe expected value of sample information using the probabilisticsensitivity analysis sample a fast nonparametric regression-basedmethod, Med. Decis. Mak., 35, 570–583, 2015b.

Tarantola, S., Gatelli, D., and Mara, T. A.: Random balance designsfor the estimation of first order global sensitivity indices, Reliab.Eng. Syst. Safe., 91, 717–727, 2006.

Vanuytrecht, E., Raes, D., and Willems, P.: Global sensitivity anal-ysis of yield output from the water productivity model, Environ.Modell. Softw., 51, 323–332, 2014.

Verrelst, J., Sabater, N., Rivera, J. P., Muñoz-Marí, J., Vicent, J.,Camps-Valls, G., and Moreno, J.: Emulation of Leaf, Canopy andAtmosphere Radiative Transfer Models for Fast Global Sensitiv-ity Analysis, Remote Sens., 8, 673–699, 2016.

Voulgarakis, A., Naik, V., Lamarque, J.-F., Shindell, D. T., Young, P.J., Prather, M. J., Wild, O., Field, R. D., Bergmann, D., Cameron-Smith, P., Cionni, I., Collins, W. J., Dalsøren, S. B., Doherty,R. M., Eyring, V., Faluvegi, G., Folberth, G. A., Horowitz, L.W., Josse, B., MacKenzie, I. A., Nagashima, T., Plummer, D.A., Righi, M., Rumbold, S. T., Stevenson, D. S., Strode, S. A.,Sudo, K., Szopa, S., and Zeng, G.: Analysis of present day andfuture OH and methane lifetime in the ACCMIP simulations, At-mos. Chem. Phys., 13, 2563–2587, https://doi.org/10.5194/acp-13-2563-2013, 2013.

Vu-Bac, N., Rafiee, R., Zhuang, X., Lahmer, T., and Rabczuk, T.:Uncertainty quantification for multiscale modeling of polymernanocomposites with correlated parameters, Composites B, 68,446–464, 2015.


https://hal.archives-ouvertes.fr/hal-00495766/document



https://doi.org/10.5194/acp-6-4427-2006

https://doi.org/10.5194/acp-6-4427-2006

https://doi.org/10.5194/acp-13-2563-2013

https://doi.org/10.5194/acp-13-2563-2013


Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J., andMorris, M. D.: Screening, predicting, and computer experiments,Technometrics, 34, 15–25, 1992.

Wild, O.: Modelling the global tropospheric ozone budget: explor-ing the variability in current models, Atmos. Chem. Phys., 7,2643–2660, https://doi.org/10.5194/acp-7-2643-2007, 2007.

Wild, O., Pochanart, P., and Akimoto, H.: Trans-Eurasian trans-port of ozone and its precursors, J. Geophys. Res.-Atmos., 109,D11302, https://doi.org/10.1029/2003JD004501, 2004.

Wild, O. and Prather, M. J.: Excitation of the primary troposphericchemical mode in a global three-dimensional model, J. Geophys.Res., 105, 24647–24660, 2000.

Wild, O., Ryan, E., O’Connor, F., Vougarakis, A., and Lee, L.: Re-ducing Uncertainty in Model Budgets of Tropospheric Ozoneand OH, Atmos.Chem. Phys., in preparation, 2018.

Wold, S., Sjöström, M., and Eriksson, L.: PLS-regression: a basictool of chemometrics, Chemom. Intell. Labor. Syst., 58, 109–130, 2001.

Wood, S. N.: Generalized additive models: an introduction with R,CRC press, New York, 2017.

Wu, J., Dhingra, R., Gambhir, M., and Remais, J. V.: Sen-sitivity analysis of infectious disease models: methods, ad-vances and their application, J. Roy. Soc. Interf., 10,https://doi.org/10.1098/rsif.2012.1018, 2013.


https://doi.org/10.5194/acp-7-2643-2007

https://doi.org/10.1029/2003JD004501

https://doi.org/10.1098/rsif.2012.1018

Documents

Fast sensitivity analysis methods for computationally ... · Sensitivity analysis is a powerful tool for understanding the behaviour of a numerical model. It allows quantiﬁcation