Comput Geosci (2011) 15:529–544DOI 10.1007/s10596-010-9222-2

ORIGINAL PAPER

Ensemble Kalman filtering for non-linear likelihoodmodels using kernel-shrinkage regression techniques

Jon Sætrom · Henning Omre

Received: 16 June 2010 / Accepted: 3 December 2010 / Published online: 19 March 2011© The Author(s) 2011. This article is published with open access at Springerlink.com

Abstract One of the major limitations of the classi-cal ensemble Kalman filter (EnKF) is the assumptionof a linear relationship between the state vector andthe observed data. Thus, the classical EnKF algorithmcan suffer from poor performance when consideringhighly non-linear and non-Gaussian likelihood models.In this paper, we have formulated the EnKF based onkernel-shrinkage regression techniques. This approachmakes it possible to handle highly non-linear likelihoodmodels efficiently. Moreover, a solution to the pre-image problem, essential in previously suggested EnKFschemes based on kernel methods, is not required. Test-ing the suggested procedure on a simple, illustrativeproblem with a non-linear likelihood model, we wereable to obtain good results when the classical EnKFfailed.

Keywords Shrinkage regression · Kernel methods ·Sequential data assimilation · Model selection

1 Introduction

In recent years, Bayesian methods have become at-tractive to use when considering geophysical inverseproblems [40]. The ensemble Kalman filter (EnKF) isa Bayesian method that provides a solution to highlynon-linear and high-dimensional spatiotemporal dataassimilation problems [1, 8]. The EnKF is defined in the

J. Sætrom (B) · H. OmreDepartment of Mathematical Sciences,Norwegian University of Science and Technology,Alfred Getz’ vei 1, 7034 Trondheim, Norwaye-mail: jonsa@math.ntnu.no

spirit of the classical Kalman filter (KF) [21], that pro-vides the analytical solution of the posterior probabilitydistribution when the prior, forward and likelihoodmodels are Gaussian and linear, termed the Gauss-linear model.

Analytical tractability of the posterior distributionwill be lost in a general model setting. Thus, we mayapply techniques such as Markov chain Monte Carlo(McMC) or rejection sampling to generate realisationsfrom the posterior distribution of interest [6]. For high-dimensional problems, however, these methods tend tobe computationally prohibitive.

The EnKF approach is based on the approximationthat the output of the forward and likelihood modelsare jointly Gaussian, with unknown mean and covari-ance. Using an ensemble of independent realisationsto estimate the model parameters empirically, ensuresthat the EnKF is consistent with the KF for Gauss-linear models as the ensemble size tends to infinity[8, 25].

As shown in Anderson [2], we can equally formu-late the classical EnKF updating scheme as a multi-variate linear regression problem, where the Kalmangain matrix defines the unknown matrix of regressioncoefficients. Hence, the classical EnKF can have poorperformance when considering highly non-linear like-lihood models. Methods such as the randomised max-imum likelihood filter (RMLF) [31] can improve onthe EnKF updating scheme for non-linear likelihoodmodels. However, the RMLF algorithm requires anoptimisation step making the method more computa-tionally demanding than the traditional EnKF. This isespecially true when considering high dimensional datasuch as time-lapse seismic. In addition, it is unclear howto use the RMLF if the error term is not additive [27].

530 Comput Geosci (2011) 15:529–544

The classical EnKF tends to underestimate the pre-diction uncertainty for small ensemble sizes [10, 17,28, 37, 48]. This can potentially lead to an ensem-ble collapsing into one single realisation. In a recentpaper, Sætrom and Omre [46], reformulated the clas-sical EnKF updating scheme using shrinkage regres-sion techniques known from multivariate statistics. Itis well known from statistical literature that the un-biased classical least squares estimator for the ma-trix of regression coefficients is not optimal in thepresence of collinear data, and can lead to severeproblems of model overfitting [12]. The purpose ofshrinkage regression techniques is therefore to replacethe classical, unbiased estimator of the unknown matrixof regression coefficients with biased alternatives hav-ing improved predictive capabilities; e.g. using dimen-sion reduction techniques on the predictor variables,or by regularising the estimated matrix of regressioncoefficients. Because the updated ensemble memberswill be coupled through the estimated Kalman gainmatrix, collinearities between the ensemble memberswill eventually occur [17]. Hence, it is not surprisingthat applying shrinkage regression techniques can leadto significant improvements compared with the classicalEnKF updating scheme, as illustrated in the examplesconsidered in Sætrom and Omre [46].

In recent years, kernel methods have become popu-lar within the field of machine learning [44, 47]. The aimof these methods is to transform data from the originalvector space into a possibly high dimensional featurespace, where we assume that the underlying model as-sumptions, such as linear dependencies in a regressionsetting, are valid. Kernel methods are frequently usedin non-linear principal component analysis (PCA) [39,41], data mining [18], classification [49] and non-linearregression [36]. Common for these methods is that thealgorithms can be reformulated through inner productsin the original space. Because we can define innerproducts in a feature space through positive definitefunctions, known as kernel functions, there is no needto generate realisations in the feature space [15, 44].

In the current paper, we extend the EnKF updatingscheme to a non-linear setting using previously definedkernel-based shrinkage regression techniques [36, 47].We demonstrate the suggested approach on a simplisticexample with a non-linear and non-Gaussian likelihoodmodel. The procedure has the same computationalcomplexity and memory requirements as the fastestimplementations of the traditional EnKF.

We are not the first to recognise the potential ofkernel methods in an EnKF setting [3, 38]. However,

the focus of these studies is to incorporate highly non-Gaussian features of the state vector into the EnKF,which require a solution to the pre-image problemof mapping the state vector from the feature spaceback to the original space. Solving this problem usingtraditional approaches requires non-linear optimisationtechniques [24, 26, 44], which can lead to high compu-tational demands. In the current study we use kernelmethods for handling non-linearities in the likelihoodmodel, which do not require this back-transformation.

2 Notation and model formulation

Throughout the paper, we use the notation x ∈ Rnx×1 to

denote that x is an nx-dimensional column vector in thereal space and xT will denote its transpose. Similarly,we will write X ∈ R

m×n to denote that X is a matrixin the real space containing m rows and n columns.Note that we will use that the same notation for bothscalars and random variables. Probability density func-tions (PDF) will be denoted by f (x), and the notationx ∼ f (x), implies that the random vector x follows thePDF f (x). Furthermore, we will denote the conditionalPDF of x given y by f (x|y). As a special case, thenotation x ∼ Gaussnx(μx, �x) will be used to denotethat x follows the nx-dimensional multivariate Gaussiandistribution with mean vector μx and covariancematrix �x.

Consider the sequence of stochastic vectors xt0, . . . ,

xtK+1; xti ∈ Rnx×1 and dt0 , . . . , dtK ; dti ∈ R

nd×1, outlinedin Fig. 1. Here, xtk denotes the state of the unknownrandom vector of interest at time step k and time tk,and similarly dtk denotes the vector of observed data.For notational convenience, we will from now on dropthe subscript tk, and simply write xk and dk. Also notethat we will for simplicity refer to x and d as the stateand observation vector respectively.

Fig. 1 Stochastic directed acyclic graph (DAG) of the modelconsidered

Comput Geosci (2011) 15:529–544 531

Let f (x0) denote the PDF of the state vector at theinitial time step. The Markov property of the directedacyclic graph (DAG) in Fig. 1 entails

f (xk+1|xk, xk−1, . . . , x0) = f (xk+1|xk), k = 0, . . . , K.

We define the PDF, f (xk+1|xk), through a known,possibly highly non-linear forward function, ω : (∈R

nx×1× ∈ Rnx×1) →∈ R

nx×1, which implies

xk+1 = ω(xk, εxk), k = 0, . . . , K. (1)

Here, εxk ∈ Rnx×1 represents random model errors or

numerical errors in the forward model, assumed to fol-low a known probability distribution. Thus, f (x0) andω(·, ·) implicitly defines the prior PDF f (x0, . . . , xK+1).We define the likelihood function, f (dk|xk), througha known, non-linear function ζ : (Rnx×1 × R

nd×1) →R

nd×1, that is,

dk = ζ (xk, εdk), k = 0, . . . , K. (2)

Again, εdk ∈ Rnd×1, represents the random observation

error following a known PDF.For notational convenience, we from now on let

xck ∼ f (xk|d0:k)

xuk ∼ f (xk|d0:k−1), k = 1, . . . , K + 1,

where d0:l denotes the sequence d0, . . . , dl for l >

0. Bayesian inversion provides a sequential solutionto the spatiotemporal forecast problem of predict-ing xc

k, for k = 1, . . . , K + 1. With such an approach,we assess the unknown vectors xc

k and xuk+1 by

sampling from the respective posterior distributions,f (xk|d0:k) and f (xk+1|d0:k). Using Bayes rule and theMarkov property of the DAG in Fig. 1, which entailsf (dk|xk, d0:(k−1)) = f (dk|xk), for k = 1, . . . , K, we get

f (xk|d0:k) ∝ f (xk|d0:(k−1)) f (dk|xk)

f (xk+1|d0:k) =∫

f (xk+1|xk) f (xk|d0:k)dxk. (3)

Generally, we only know the conditional distribu-tions defined in Eq. 3 up to an unknown normalisingconstant. One possibility is to use computationallydemanding techniques such as McMC or rejection sam-pling to generate realisation from the correct poste-rior distribution [6]. However, for applications such aspetroleum reservoir evaluation, these techniques arecomputationally prohibitive [8]. An approximate so-lution can be obtained by assuming that xu

k and dk

follows a distribution that ensures analytical tractabil-ity of f (xk|d0:k), such as the Gaussian. These model

assumptions are equivalent to those made in theEnKF [7].

3 Classical ensemble Kalman filter

Let

xu(i)k = ω

(xc(i)

k−1, ε(i)xk−1

)

and

d(i)k = ζ

(xu(i)

k , ε(i)dk

),

for i = 1, . . . , ne, and define Xk =[xu(1)

k , . . . , xu(ne)

k

]∈

Rnx×ne and Dk =

[d(1)

k , . . . , d(ne)

k

]∈ R

nd×ne as the stateensemble and data ensembles matrices respectively.For notational convenience, we will from now on omitthe subscript k because the focus will be on a single timestep.

If we assume that the joint distribution of (xu, d) isGaussian, a classical updating scheme for each ensem-ble member would be:

xc(i) = xu(i) + K̂(d − d(i)), (4)

where

K̂ = X H DT (DH DT)−1 ∈ R

nx×nd . (5)

We refer to this as the classical EnKF updating scheme,where we denote the estimated Kalman gain matrix byK̂. Here,

H = I − 1

ne11T ∈ R

ne×ne (6)

is the idempotent centring matrix, where I is the iden-tity matrix, and 1 is a vector with each entry equalto one, both having proper dimensions. Under theGaussian assumption stated above, xc(i)

k will tend to-wards a realisation from the Gaussian posterior distrib-ution f (xk|d0:k) as ne → ∞.

From multivariate statistical theory, we know thatthe estimated Kalman gain matrix is equal to the leastsquares estimate of the matrix of regression coefficientsin a multivariate linear regression setting [42]:

K̂ = arg minK

tr{(X H − K DH)(X H − K DH)T}

, (7)

where tr{·} denotes the trace operator. Hence, it is notsurprising that the classical EnKF can perform poorlyfor highly non-linear functions in the likelihood model.

Note that for situations where nd ≥ ne, the matrixDH DT in Eq. 5 will be singular, meaning that its

532 Comput Geosci (2011) 15:529–544

inverse does not exist. To avoid this problem wecan add a positive definite matrix, (ne − 1)�r, whichcorresponds to adding a regularisation term, (ne −1)tr{K�r KT}, to the objective function in Eq. 7. Thisgives

K̂ = X H DT (DH DT + (ne − 1)�r

)−1, (8)

which is an extension of the standard EnKF updatingscheme to non-linear likelihood models, similar to theapproach in [8, Appendix A.2]. Assuming the followinglikelihood model, d = ζ (xu) + εd , where the indepen-dent Gaussian noise term has zero mean and covari-ance, �r, the Kalman gain estimate defined in Eq. 8is natural to consider. If we replace the data ensemblematrix used in Eq. 8, with an alternative data ensemblematrix, D̃ = [ζ (xu(1)), . . . , ζ (xu(ne))], the estimator

K̂ = X H D̃T(

D̃H D̃T + (ne − 1)�r

)−1, (9)

will indeed be consistent with the estimator definedin Eq. 5. This follows because the data ensemble ma-trix can be split into two parts, D = D̃ + E, whereE = [ε(1)

d , . . . , ε(ne)

d ] is error perturbation ensemble ma-trix. However, in the general model setting with d =ζ (x, εd), the two estimators will not be consistent.

Another potential problem with the classical leastsquared estimate of the Kalman gain is modeloverfitting. This is especially true in the presence ofcollinear data [9]. Because we couple the updated en-semble members through the estimated Kalman gainmatrix, they can become increasingly collinear withtime [28]. This can potentially lead to an ensemblecollapsing into a single realisation, and certainly lead toan underestimation of the prediction uncertainty [46].

Fortunately, we can handle the problems describedabove efficiently using kernel shrinkage regressiontechniques known from multivariate non-linear regres-sion, which we will consider next.

4 Kernel-shrinkage regression

To motivate the use of kernel methods, we startthis section with a probabilistic discussion of a non-linear regression problem with multivariate predictorvariables:

x = γ (d) + εx|d . (10)

Here, x is a univariate, centred response variable; d isa centred multivariate predictor variable; γ (d) is the

non-linear regression function; and εx|d is a univari-ate Gaussian error term having zero mean and vari-ance σ 2

x|d . For notational simplicity we only considerone-dimensional response variables, because it can beshown [12] that we under certain assumptions obtaincorresponding solutions for each component in themultivariate case.

Following Williams [50], let the non-linear functionγ (·) be decomposed into L terms

x = βTϕ(d) + εx|d,

where β = [β1, . . . , βL]T ∼ GaussL(0, I) is an un-known random vector. Here 0 is the zero vectorof proper dimensions, and the L-dimensional vectorϕ(d) = [ϕ1(d), . . . , ϕL(d)]T , is a collection of knownlink functions. Two examples of link functions are

ϕk(d) = a0k + a1k dk + a2k d2k

ϕk(d) = σϕk exp

{−

nd∑l=1

(dl − τk,l)2

�2k,l

}, (11)

k = 1, . . . , L, where ai,k, σϕk , τk,l and �k,l are modelparameters. Under the assumption of independent ob-servation errors for εx|d this entails:

E[x] = 0

Cov(x, x′) = ϕ(d)Tϕ(d ′) + δ(d, d ′)σ 2x|d,

with δ(·, ·) being the Dirac function taking value onewhen the arguments are identical and zero otherwise.

Consider a set of centred realisations {(x(i), d(i)), i =1, . . . , ne} and define the corresponding univariate, cen-tred state vector x̃ = [x(1), . . . , x(ne)] ∈ R

ne×1 with asso-ciated centred data ensemble matrix D. In addition, letd∗ be a new, centred data vector, with unknown cen-tred state variable x∗. Under the Gaussian assumptionabove we have[

x∗

x̃

]∼ Gauss1+ne

([00

],

[Cov(x∗) Cov(x∗, x̃)

Cov(x̃, x∗) Cov(x̃)

]),

where Cov(x∗) = ϕ(d∗)Tϕ(d∗) + σ 2x|d , Cov(x∗, x̃) =

ϕ(d∗)T� and Cov(x̃) = �T� + σ 2x|d I, with � =

[ϕ(d(1)), . . . , ϕ(d(ne))] ∈ RL×ne . The conditional expec-

tation of x∗ given x̃, E[x∗|x̃], minimises the meansquared prediction error (MSPE), E[(x∗ − g(x̃))2], forany function g : R

ne → R [32]. Hence,

x̂∗ = x̃T(�T� + σ 2

x|d I)−1

�Tϕ(d∗) (12)

is the optimal predictor of x∗ in terms of the MSPE.

Comput Geosci (2011) 15:529–544 533

An alternative formulation of the non-linear regres-sion problem is to parametrise γ (d) by a Gaussianrandom field (GRF), as defined in Appendix A,with mean E[γ (d)] = 0, and covariance functionCov

(γ (d), γ (d ′)

) = c(d, d ′). Using the definition of aGRF together with well-known results from multivari-ate Gaussian theory, the predictive mean of x∗ given x̃is:

E[x∗|x̃] = x̃T(

C + σ 2x|d I

)−1c∗ (13)

with c∗ = [c(d(1), d∗), . . . , c(d(ne), d∗)]T ∈ Rne×1, and

C =⎡⎢⎣

c(d(1), d(1)) . . . c(d(1), d(ne))...

. . ....

c(d(ne), d(1)) . . . c(d(ne), d(ne))

⎤⎥⎦ ∈ R

ne×ne .

Comparing Eqs. 12 and 13, we see that there is adual formulation of the non-linear regression problem,where we can interpret inner products between vectorsϕ(·) ∈ R

L×1 as covariance functions c(·, ·) of a GRF.For the link-function defined in Eq. 11 this can berealised by letting L → ∞ and selecting �k,l = � forall k and l, which entails [11]:

c(d, d ′) = ϕ(d)Tϕ(d ′) = σ 2ϕ exp

{− 1

2�‖d − d ′‖2

2

}(14)

which we recognise as the second-order exponential co-variance function. The assumption that we can describeinner products between vectors ϕ(d) and ϕ(d ′) throughsymmetric, positive definite functions, c(d, d ′), is thefoundation of kernel methods known from the machinelearning literature [44].

4.1 Kernel methods

We now generalise the mapping ϕ(·), and let ϕ : Rnd →

F , where F is some unspecified inner product space[52], which we for simplicity will refer to as the featurespace. In the non-linear regression setting this entailstransforming the predictor variable, d, into a featurespace where the linear relationship with the responsevariable, x, is valid, as described in Fig. 2.

In Eq. 12, we express x̂∗ through inner productsin the feature space, which we can equally describethrough values of the covariance function, c(·, ·), un-der the GRF assumption. In general, this will holdfor any symmetric, positive definite function, c(·, ·),also known as kernel functions [44]. In the machinelearning literature, this is known as the “kernel-trick”[44], which in practise means that we can reformulateany algorithm involving inner products in the input

Fig. 2 Transformation of d into a feature space F , where thereis a linear relationship with the response x

space (here Rnd ) into a feature space, F , using ker-

nel functions. The formal conditions for which the as-sumption, c(d, d ′) = 〈ϕ(d), ϕ(d ′)〉 holds are describedin Hoffmann et al. [15], namely symmetry and positivedefiniteness. The interested reader can find furtherdetails in Smola and Schölkopf [44] or Taylor andCristianini [47].

Sætrom and Omre [46], reformulated the classicalEnKF updating scheme using principal component re-gression (PCR) [16] and partial least squares regression(PLSR) [51], known from shrinkage regression. Bothmethods can be transformed into a non-linear settingusing kernelised versions; see Rosipal et al. [35] andRosipal and Trejo [36] for PCR and PLSR, respectively.Common to the kernel versions of these two shrinkageregression techniques is that we perform a dimensionreduction in the feature space. Moreover, given a newobservation vector, d∗, we predict the state variable, x∗,by evaluating the kernel function. Hence, the methodsdoes not require an inverse mapping, avoiding the pre-image problem occurring when using the kernel ap-proach on the state variables [3, 38].

4.2 Comments

An early reference to the Bayesian formulation ofthe non-linear regression problem is O’Hagan [30].However, the connection with kernel methods knownfrom the machine learning literature was not realiseduntil recently [33, 50]. Moreover, readers familiar withGeostatistics will surely recognise the connection be-tween the probabilistic formulation above and a gen-eralisation of kriging using non-stationary covariancemodels [5, 19].

The decoupling of the state vector into univariatecomponents, as done in the decomposition above, istheoretically only valid if the elements in the stochasticerror term, εx|d , are independent, which implies that the

534 Comput Geosci (2011) 15:529–544

covariance �εx|d is proportional to the identity matrix,I. Note, however, that the methods considered in thispaper will not be affected by this model assumptionbecause we are solving the regression problem using aleast squares approach. This entails that the obtainedregression models do not depend on the model para-meter �εx|d [12].

5 Ensemble Kalman filtering using kernel-shrinkageregression

We will now present the kernelised versions of theEnKF updating scheme based on three common shrink-age regression techniques; ridge regression (RR) [14],PCR and PLSR. However, we will only consider kernelRR in detail. This is motivated because kernel RRis connected to the probabilistic solution of the non-linear regression problem discussed above; details aregiven below. The interested reader can find a thoroughdescription of the PCR and PLSR shrinkage regressiontechniques in Sætrom and Omre [46].

For simplicity, we define kernel matrices C∗ =[c∗, . . . , c∗] ∈ R

ne×ne , with c∗ = [c(d(1), d∗), . . . , c(d(ne),

d∗)]T ∈ Rne×1, and

C =⎡⎢⎣

c(d(1), d(1)

). . . c

(d(1), d(ne)

)...

. . ....

c(d(ne), d(1)

). . . c

(d(ne), d(ne)

)

⎤⎥⎦ . (15)

For all three methods, we find the estimated Kalmangain matrix by solving the following multivariate linearregression problem in a feature space, F :

x = Kϕ(d) + εx|d,

where εx|d ∈ Rnx×1 represents the regression model er-

ror. Similar to above, � = [ϕ(d(1)), . . . , ϕ(d(ne))], andwe assume that all vectors and ensemble matrices arecentred, unless otherwise stated.

5.1 Kernel ridge regression

Kernel RR is a regularisation method where we selectthe estimated regression coefficients by minimising themean squared error with additional constraints, that is:

K̂KerRR = arg minK

{tr{(X − K�)(X − K�)T}

+ ξ tr{KKT}} .

Solving this minimisation problem analytically gives[42]:

K̂KerRR = X�T (��T + ξ I

)−1

= X(�T� + ξ I

)−1�T .

Thus, we obtain kernel RR predictions for an unknownx∗, with associated d∗, based on

x∗ = X (C + ξ I)−1 c∗ (16)

Comparing this expression with Eq. 13, we see thatthe two predictors, x∗ and E[x∗|X] are identical forσ 2

x|d = ξ . This follows because each variable x∗i , for

i = 1, . . . , nx, can be computed independently. Thus,applying kernel RR in an EnKF setting leads to thefollowing updating scheme:

Xc = Xu + Xu H (HCH + ξ I)−1 H(C∗ − C). (17)

Here, Xc and Xu are the state ensemble matrices, con-ditioned and unconditioned respectively and HCH isthe centred kernel matrix [41], with the centring matrix,H, defined in Eq. 6.

5.2 Kernel principal component regression

PCR is based on the assumption that most of thevariability in the predictor variables can be explainedthrough a small set of random variables, termed prin-cipal components. The estimated matrix of regressioncoefficients is then constructed using the principal com-ponents as predictor variables. Hence, we define amatrix of regression coefficients in a reduced orderspace. Kernel PCR follows directly by applying kernelPCA [35, 41] to ϕ(d). The resulting expression for theestimated Kalman gain matrix is

K̂KerPCR = X Ep−1p ET

p �T . (18)

Here Ep ∈ Rne×p contains the p eigenvectors of the

centred kernel matrix, HCH, with the p largest cor-responding eigenvalues given in the diagonal matrix,p ∈ R

p×p. The following expression then forms anEnKF updating scheme based on kernel PCR:

Xc = Xu + Xu Ep−1p ET

p (C∗ − C). (19)

5.3 Kernel partial least squares regression

Similar to PCR, PLSR applies dimension reduc-tion techniques to the predictor variables. The maindifference between the two methods is that, PLSRuses the information available from both the response

Comput Geosci (2011) 15:529–544 535

and predictor variables when projecting into the p-dimensional subspace, which entails that PLSR is basedon a supervised dimension reduction technique. Thekernelised version of the PLSR algorithm follows fromthe algorithm presented in Ränner et al. [34]. Thisresults in the following estimate of the Kalman gainmatrix:

K̂KerPLSR = XT A−1W T�T .

Here, A = (W T CT

) ∈ Rp×p, with latent variables

T = [t1, . . . , tp] ∈ Rne×p and W = [w1, . . . , wp] ∈ R

ne×p

given by solving sequentially for i = 1, . . . , p:

[ti = H�Tψi

wi = H XTυi

]←

⎧⎨⎩

maxψi,υi{υTi X H�Tψi}

‖ψi‖2 = 1, ‖υi‖2 = 1tTi t j = 0, for all j < i.

Ränner et al. [34] outline an efficient procedure forsolving this problem when nx and nd are larger than ne.This gives the following EnKF updating scheme basedon kernel PLSR:

Xc = Xu + XuT A−1W T(C∗ − C), (20)

with A defined above. Note that centring of the en-semble matrices is unnecessary because of the identitiesT = HT and W = HW [46].

5.4 Comments

Although the three shrinkage regression techniquespresented above, and their kernelised versions, all re-duce the problems caused by collinearities in the dataensemble, they accomplish this differently. Note thatcollinearities in the data ensemble will result in smalleigenvalues in the estimated data covariance matrix,�̂d , or the corresponding centred kernel matrix, HCH,which can result in large weights in the respectiveKalman gain matrices. Whilst the RR technique re-duces the weight caused by smallest eigenvalues byadding a positive constant to all eigenvalues, the PCRtechnique eliminates small eigenvalues, whilst retainingthe p dominant. The PLSR technique works in a similarmanner as the PCR, although the analysis is slightlymore complicated (see Hastie et al. [12] for a detaileddescription). However, it has been noted that becausethe PLSR technique uses the information from boththe state and data ensemble matrices in the dimensionreduction, a small number of components, p, is oftenrequired in PLSR compared with PCR [13, 20].

It should be noted that the optimality of the differentshrinkage regression techniques appears to be problemdependent. That is, whilst RR leads to the smallest

prediction error in one study, PCR or PLSR can beoptimal in other studies. It is therefore advisable totest the predictive performance of the different meth-ods based on the prior and likelihood model, beforeselecting which method to use in an EnKF setting.Finally it is important to note that the performance ofall three schemes are highly dependent on the modelhyperparameters used. Thus, schemes which enablesan automatic selection of these model parameters arecalled for.

5.5 Model hyperparameter selection

The results obtained using the kernel-shrinkage regres-sion techniques described above, will in general dependon one or more hyperparameters, θ . For kernel RRthis involves selecting the size of the regularisationparameter, ξ , whilst for kernel PCR and PLSR thedimension of the reduced order space, p, is required. Inaddition, we need to select the kernel function, c(·, ·),which implicitly defines ϕ(·). This can be challengingif all symmetric, positive definite kernel functions areconsidered.

The usual approach for selecting c(·, ·) is to consideronly certain families of kernel functions. An example istranslation and rotation invariant functions c(d, d ′) =c(‖d − d ′‖2

2), referred to as radial basis function (RBF)kernels, where ‖ · ‖2 is the Euclidean norm [44]. Thesecond order exponential kernel function, defined inEq. 11, is an example of such a kernel function. An-other example of a kernel function is the polynomial,defined as c(d, d ′) = (1 + dT d ′)ν, ν > 1, which entailsthat ϕ(·) represents polynomials. Common for most ofthe kernel functions in the literature, however, is thatwe only need to specify one or two hyperparameters.

To avoid overfitting the regression model to thedata, we can apply cross-validation (CV), as discussedin Sætrom and Omre [46]. CV works by sequentiallysplitting the state vector and data ensembles into train-ing ensemble matrices, (XTrain, DTrain), used for modelestimation, and test ensembles matrices, (XTest, DTest),used for model validation. Using the training ensemblematrices to estimate the Kalman gain matrix for modelparameters θ , K̂Train(θ), we can predict the state vectorsbased on the test data, X̂Test = K̂Train(θ)DTest. Com-puting the sum of squares of the mismatch betweenX̂Test and XTest, referred to as the predictive error sumof squares (PRESS) statistic, can be used to measureof the predictive power for the chosen model. Note,however, that it is straightforward to use other objec-tive functions than the sum of squares. We can then

536 Comput Geosci (2011) 15:529–544

select the model hyperparameters by minimising thePRESS statistic for different values of θ , as discussedbelow. A pseudo-code describing the workflow of theEnKF updating scheme based on kernel PCR, usingm-fold CV to select the model parameters is given inAlgorithm 1. Note that we here use the notation Ii todenote the set of indices for members of test ensemblei. A similar workflow can be used for the RR and PLSRtechniques as well.

The model parameters, θ , will for most applicationsof kernel-shrinkage regression contain two parameters,which can be both discrete and continuous. Alternativemethods for solving this problem are gradient basedoptimisation techniques [29], or stochastic optimisationtechniques such as particle swarm optimisation [22] andsimulated annealing [23]. In practise, however, it is of-ten sufficient to consider a limited number discrete val-ues for the continuous model parameters, for example� ∈ {1, 2, . . . , 20} for the second order exponential co-variance function, rather than searching for the globaloptimum [44]. Hence, we do not necessarily suffer fromhigh computational demands if we select the modelparameters based on an exhaustive search rather thansophisticated optimising schemes. Also note that forapplications of the EnKF where it is extremely timeconsuming to evaluate the forward model ω(·), suchas petroleum reservoir characterisation, the use of CVfor model parameter selection will not increase the

total computational time significantly. In addition, weemphasise that the CV scheme is straightforward to runin parallel. A discussion regarding the computationalproperties of the EnKF updating scheme based onkernel-shrinkage regression techniques can be found inAppendix B.

Finally, because the likelihood model is known, se-lection of the kernel function is easier in this setting,compared with non-linear regression problems wherethe relation between d and x is unknown. An alterna-tive approach is therefore to select the kernel functionbased on prior knowledge of the state vector and thelikelihood model. That is, we select the hyperpara-meters of the kernel function based on the predictivepower of the non-linear regression model created usingrealisations from the initial prior model.

6 Empirical study

We define the state vector xk ∈ R100×1, k = 0, . . . , 19 on

a (10 × 10) regular grid domain. Here, xi, j,k denotes thevalue of the state vector at location (i, j), i = 1, . . . , 10,j = 1, . . . , 10 and time step k. We assume that the statevector is static, meaning xk+1 = xk, k = 0, . . . , 19, witha reference model generated from the Gaussian priormodel:

xTrue ∼ Gaussnx(μx, �x).

Here, μx = 5 × 1, and we construct �x based on anexponential covariance function,

Cov(xi, j,0, xl,m,0) = exp{−3( )1.2

}, (21)

where

Δ =√(

Δx

lx

)2

+(

Δy

ly

)2

.

The range parameters lx and ly are one and 10,respectively.

We use the non-linear likelihood model:

dk = ζ (x, εd), (22)

to connect the observations dk to the state vector.Here each element ζi, j,k = g(xi, j,k, 5) + σdε

2i, j,k, with g(·)

defined as:

g(x, β) =⎧⎨⎩

sin(|x − β|) cos(d)

|x − β| , x �= β

1, x = β.

, (23)

and εi, j,k ∼ Gauss1(0, 1). Hence, we have an addi-tive error term εd/σd ∼ χ2

1 , where χ2ν denotes the

Comput Geosci (2011) 15:529–544 537

χ2-distribution with ν-degrees of freedom [4], whichmakes the likelihood function both non-Gaussian andnon-linear. Here σd = 0.25, which implies that the meanand variance of the error term are equal to 0.250 and0.125, respectively. Figure 3 shows an image plot of thereference state vector. The scatter plot of xTrue and thecorresponding observed data at the initial time step,d0, in Fig. 4, displays the non-linear structure of thelikelihood model. At each time step k = 0, . . . , 19, wemake new observations, dk.

6.1 Non-linear regression

To demonstrate the effect of the suggested kernel re-gression techniques we initially consider a univariatenon-linear regression problem, with a bivariate predic-tor variable, based on the function defined in Eq. 23 asfollows: Define the date ensemble matrix, D, throughne = 91 uniformly spaced points, d ∈ R

2×1, with d1 andd2 having values between −4 and 4. Assume that ateach of these 91 locations, we observe values of thefunction,

x = γ (d) + εx|d = 1 + g(d1, 0) + g(d2, 0) + εx|d,

where the noise term, εx|d , is Gaussian having zeromean and standard deviation 0.5. Figures 5 and 6 dis-play the reference solution and the observed data.

Using the ensemble matrices D and X, the task is tofit a regression model based on the classical linear leastsquares approach and the three kernel-shrinkage re-gression models discussed above. Because the Gaussiankernel function, described in Eq. 14, is a robust choicewhen no prior information is available regarding the

Fig. 3 Reference state vector used in the empirical case studygenerated from a Gaussian prior distribution with an anisotropiccovariance function

1 2 3 4 5 6 7–1

–0.5

0

0.5

1

1.5

2

Fig. 4 Scatter plot between the elements of xTrue and cor-responding observations, do, based on the non-linear, non-Gaussian likelihood model

data [43, 45], we use it to map d into the featurespace F . To select the scaling factor in the kernelfunction and shrinkage factor for the three regressionmethods, we apply 10-fold CV. Figure 7 display the re-sults. Note that because the prior and likelihood modelsare fully specified, another alternative is to tailor thekernel function based on this known prior information.How this task can be accomplished in an automaticmanner is, however, somewhat unclear and is a topic forfurther research. Hence, we apply the Gaussian kernelfunction for the case studies considered in this paper,even if this selection might be sub-optimal.

–4–2

02

4

–4

–2

0

2

40

0.5

1

1.5

2

2.5

3

Fig. 5 Reference solution for the non-linear regression problemconsidered, x = γ (d), for locations, d in the plane

538 Comput Geosci (2011) 15:529–544

−4−2

02

4

−4

−2

0

2

40

0.5

1

1.5

2

2.5

3

Fig. 6 Observed data of the function x = γ (d) + εx|d at 81 uni-formly spaced locations d in the plane

As expected, the plane resulting from the linear leastsquares method, does not provide a good prediction ofthe non-linear function. The three kernel methods, onthe other hand, are able to capture the non-linear trendof the function, thus providing reasonable predictions.Figure 8 display the absolute deviance between thereference and predicted solution for the four methods.Except for the larger error occurring at the origin forthe kernel RR method, the deviance is within one stan-dard deviation of the observation error at all predicted

–50

5

–5

0

5–5

0

5

10

LSQ

–50

5

–5

0

50

1

2

3

KerRR

–50

5

–5

0

50

1

2

3

KerPCR

–50

5

–5

0

50

1

2

3

KerPLSR

Fig. 7 Predicted outcome based on least squares linear regres-sion (LSQ), kernel RR (KerRR), kernel PCR (KerPCR) andkernel PLSR (KerPLSR); 10-fold CV was used to select the priorhyperparameters θ

Fig. 8 Absolute deviance between the true solution and the pre-dicted outcome based on least squares linear regression (LSQ),kernel RR (KerRR), kernel PCR (KerPCR) and kernel PLSR(KerPLSR). 10-fold CV was used to select the prior hyperpara-meters θ

locations using the three kernel methods. The maindifference between the PCR and PLSR approaches is,however, that the former requires p = 13 components,whilst the latter only requires p = 1. Note that thisbehaviour is in accordance with the discussion above,because PLSR uses information from both the state-and data vector in the dimension reduction.

6.2 Results

Based on the results seen in the non-linear regressionbased on the likelihood model, we proceed using theEnKF based on kernel PCR and PLSR techniques onthe filtering problem with the Gaussian kernel func-tion defined in Eq. 14. We use the following samplingscheme to generate the initial ensemble with ne = 100:

xu(1) = 10 × 1

xu(2), . . . , xu(25) ∼ Gaussnx

(1

1.5μx, 2�x

)

xu(26), . . . , xu(50) ∼ Gaussnx

(1

2μx, 2�x

)

xu(51), . . . , xu(75) ∼ Gaussnx (2μx, 2�x)

xu(76), . . . , xu(100) ∼ Gaussnx (1.5μx, 2�x) ,

with μx and �x defined above. Hence, there is a highuncertainty regarding the mean of the true underlyingdistribution. The purpose of this study is to assimi-

Comput Geosci (2011) 15:529–544 539

late observed data using the standard EnKF updat-ing scheme for non-linear observations [8, AppendixA.2] and the EnKF updating schemes based onkernel-shrinkage regression. Because nd = ne, �̂d willbe singular. We therefore add a positive definite regu-larisation term before inverting the matrix, as explainedin Section 3, with �r = 2σ 2

d I.When selecting prior hyperparameters in the kernel-

shrinkage regression techniques, θk, we consider thefollowing two approaches:

– Automatic: θ = (σ 2, p) selected based on 10-foldCV at each time step, minimising the PRESSstatistic.

– Supervised: θ = (σ 2, p) selected based is based on10-fold CV at the initial time step, minimising thePRESS statistic, and remain fixed for all time stepsk = 0, . . . , 19.

Hence, we consider five different EnKF updatingschemes in total:

– Classical EnKF: The non-linear formulation ofthe EnKF updating scheme using the estimatedKalman gain matrix given in Eq. 9, assuming aGuassian additive noise term: d = ζ (x) + εd , whereεd ∼ Gaussnd(0, 2σ 2

d I). This implies that the covari-ance of the noise term, �r, is equal to the truevariance of the likelihood model [4], although wedo not account for the non-zero mean.

– EnKF-KerPLSR1: EnKF based on the kernelisedPLSR method. The dimension of the reduced orderspace, and the scaling parameter in the Gaussiankernel function selected based on the automaticapproach

– EnKF-KerPCR1: EnKF based on the kernelisedPCR method. The dimension of the reduced order

82 84 86 88 90 92 94 96 98 1000

5

10

15

(a) Classical EnKF

82 84 86 88 90 92 94 96 98 1000

5

10

15

(b) EnKF-KerPLSR1

82 84 86 88 90 92 94 96 98 1000

5

10

15

(c) EnKF-KerPCR1

82 84 86 88 90 92 94 96 98 1000

5

10

15

(d) EnKF-KerPLSR2

82 84 86 88 90 92 94 96 98 1000

5

10

15

(e) EnKF-KerPCR2

Fig. 9 Results obtained when running five different EnKF up-dating schemes on a problem with a highly non-linear likelihoodmodel. The figure displays, for grid nodes 81 through 100, thereference xTrue

10 (solid), the ensemble average (dotted) and the es-

timated 95% confidence bounds of the prediction interval (solid,light grey) at the final time step. The ensemble mean at the initialtime step is shown as the dashed, dark grey line

540 Comput Geosci (2011) 15:529–544

space, and the scaling parameter in the Gaussiankernel function selected based on the automaticapproach

– EnKF-KerPLSR2: EnKF based on the kernelisedPLSR method. The dimension of the reduced orderspace, and the scaling parameter in the Gaussiankernel function selected based on the supervisedapproach.

– EnKF-KerPCR2: EnKF based on the kernelisedPCR method. The dimension of the reduced orderspace, and the scaling parameter in the Gaussiankernel function selected based on the supervisedapproach

Figure 9a–e, displays the results obtained in grid nodes81 through 100 using the five different EnKF updatingschemes. The results are similar for the remaining lo-cations, and are therefore not included. Note that theinitial ensembles are identical for all five schemes.

As we can see from Fig. 9a, the classical EnKF is notable to get a good representation of the state vector at

the final updating step. The estimated posterior mean isfarther from the reference state vector than at the initialtime step and the ensemble has collapsed into a singlerealisation. Because we are using a linear updatingscheme on a non-linear likelihood model, we expectthat the ensemble mean is missing the reference statevector. However, it is more troubling that we are notable to obtain estimates of the prediction uncertainty.Ideally, the estimated prediction interval based on theupdated ensemble, should cover 95% of the referencestate vector. In this case, however, the coverage is 0%.

The EnKF-KerPLSR1 method is able to obtain rea-sonable estimates of the reference state vector, xTrue.The estimated posterior mean is centred around xTrue,and the estimated prediction interval is giving a gooddescription of the solution uncertainty. The EnKF-KerPCR1, EnKF-KerPLSR2 and EnKF-KerPCR2 up-dating schemes have a similar behaviour, as shown inFig. 9c–e.

Figure 10 contains two realisations of the state vec-tor in the initial ensemble and the corresponding re-

Fig. 10 Two realisations andthe estimated ensemble meanfor the initial ensemble, theclassical EnKF updatingscheme (C-EnKF) and thefour kernel-shrinkageregression techniques

Comput Geosci (2011) 15:529–544 541

Table 1 Scaled (1/ne) total sum of squares (TSS) of the esti-mated posterior mean to the reference solution, and coverage ofthe reference solution in the estimated 95% prediction intervalsbased on 100 different initial ensembles

ne Scheme TSS Coverage(%)

100 No updating 70.6 100.0100 Classical EnKF 311.5 6.1100 EnKF-KerPLSR1 9.3 92.2100 EnKF-KerPCR1 9.0 96.6100 EnKF-KerPLSR2 7.5 93.6100 EnKF-KerPCR2 6.2 95.6100 EnKF-PLSR1 60.5 19.3

alisations after the final updating step, using the fivedifferent EnKF updating schemes. We also display theensemble mean. At the initial time step, the realisationsand the ensemble mean do not resemble the referencestate vector, as expected from the prior distribution de-scribed above. Again, we notice that the realisations ob-tained using the classical EnKF scheme has collapsed,and are farther from xTrue, than they were initially. Thefour kernel-shrinkage regression techniques, on the

other hand, appear to give a good representation ofthe reference state vector. We especially note that theanisotropic behaviour in xTrue is captured in both therealisations and the ensemble mean for all of the fourschemes based on kernel-shrinkage regression.

It is interesting to note that the results obtainedusing the automatic and supervised methods to select θ

produces similar results. For the supervised parameterselection approach θ was selected as: θPCR = (10, 5)

and θPLSR = (10, 1), whilst for the CV based selec-tion scheme θPCR ∈ {[1, 7](5), [5, 30](20)} and θPLSR ∈{[1, 5](1), [5, 30](20)}. Here the notation [1, 5](5) is usedto denote the smallest, largest and median value of theselected parameters respectively.

To quantify the performance of the five updatingschemes the model is rerun 100 times using differentinitial ensembles. We then compute the scaled to-tal sum of squares (TSS), given as 1/ne

∑nei=1 ‖μ̂xc −

xTrue‖22, and the coverage of the reference solution

within the estimated 95% prediction intervals. Theresults are summarised in Table 1. Here we have in-cluded the results obtained using PLSR regression inthe Euclidean space, rather than the feature space,

0 5 10 15 202

3

4

5

6

7

8

9

Classical EnKFEnKF–KerPLSR1EnKF–KerPCR1EnKF–KerPLSR2EnKF–KerPCR2EnKF–PLSR1

(a) RSS

0 5 10 15 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Classical EnKFEnKF–KerPLSR1EnKF–KerPCR1EnKF–KerPLSR2EnKF–KerPCR2EnKF–PLSR1

(b) Coverage

Fig. 11 Scaled (1/ne) residual sum of squares (RSS), for the forecasted state vector ensemble members and the reference solution, andcoverage of the reference solution in the estimated 95% prediction intervals as a function of time steps k

542 Comput Geosci (2011) 15:529–544

selecting the subspace dimension p using 10-fold CV(EnKF-PLSR1). As we can see from this table, the TSSincrease with 412% from the initial updating step whenusing the classical EnKF updating scheme. On average,the prediction interval only covers 6.2% of the refer-ence state vector. Note, however, that the coverage forthe majority of the reruns is zero percent.

Using the kernelised shrinkage regression tech-niques reduces the TSS with between 87% and 91%,with the largest decrease when we use the supervisedparameter selection scheme. This suggests that thePRESS statistic is not the optimal measure of thegoodness-of-fit for this model. Replacing the PRESSstatistic with alternative measures, is a topic for futureresearch. The estimated mean coverage is close to thetheoretical value of 95% for all four schemes.

Contrary to the classical EnKF updating scheme,using the EnKF-PLSR1 scheme does not lead to anincrease in the TSS compared with the initial ensemble.As explained in Sætrom and Omre [46], we expectto see this behaviour because we reduce the problemof regression model overfitting caused by collinearensemble members, when using shrinkage regressiontechniques. However, the estimated prediction intervalis not able to capture the reference state vector, whichwe expect when using a linear updating scheme on ahighly non-linear likelihood model.

Figure 11a, b, further illustrates the negative effectof model overfitting. Here, the scaled residual sum ofsquares (RSS) between the ensemble members and thereference solution, 1/ne

∑nei=1 ‖xc(i) − xTrue‖2

2, and thecoverage as a function of k = 0, . . . , 19 are displayed.As we can see from these figures, the RSS for theclassical EnKF is increasing rapidly for the first two up-dating steps. At the same time, the coverage is rapidlydecreasing and after two assimilation steps it is down tozero percent. A similar trend can be seen for the EnKF-PLSR1 updating scheme. However, the effect of modeloverfitting is not as prominent as for the classical EnKF.This illustrates the usefulness of applying shrinkageregression techniques in an EnKF updating scheme.

For the kernelised shrinkage regression techniques,the behaviour of the RSS and coverage is similar forall of the four updating schemes considered. Initially,the RSS decrease rapidly before it stabilises at the latertime steps. Similarly, the coverage is slowly convergingtowards the expected value of 95%, where it appearsto stabilise. In addition, note that we are better ableto preserve the spread of the updated ensemble mem-bers using the kernel-shrinkage regression techniquescompared with the EnKF-PLSR1 updating scheme. Webelieve that the additional non-linearities introducedwhen mapping the data vector into the feature space

is the cause of this behaviour. Hence, we reduce thecollinearities between the updated ensemble members,which potentially can lead to model overfitting.

7 Conclusions

We have formulated an EnKF updating scheme basedon kernel-shrinkage regression techniques to handlehighly non-linear and non-Gaussian likelihood mod-els. Contrary to previously suggested EnKF updatingschemes based on kernel methods, the approach doesnot require solving a pre-image problem. Moreover, thecomputational complexity is equal to the fastest EnKFalgorithms previously suggested.

We presented kernel regression as a natural exten-sion of the classical EnKF to the non-linear case usingGRF. Under the assumption that the response variableis a GRF, we obtained a prediction scheme definedthrough the inner product between data vectors in thefeature space. We can equally describe these innerproducts through symmetric, positive definite kernelfunctions in the Euclidean space, which corresponds tocovariance functions for a GRF. Hence, we gave anextension of the Bayesian formulation of the classicalEnKF to a non-linear setting, from which kernel RRis a special case. In addition, we considered two addi-tional kernel shrinkage regression techniques based ondimension reduction, namely kernel PCR and PLSR.

We evaluated the performance of the three kernel-shrinkage regression techniques for a non-linear regres-sion problem. Here we used a CV scheme to selectthe prior hyperparameters of the regression models.When we applied the estimated models for predictionpurposes, we obtained similar results for the kernelPCR and PLSR methods. However, the dimension ofthe respective reduced order models was smaller for thekernel PLSR, because this method uses both the predic-tor and response variables in the dimension reduction.Kernel RR gave slightly larger errors in the predictions.

EnKF updating schemes based on kernel PCR andPLSR were further tested on a hidden Markov modelwith a non-linear, non-Gaussian likelihood model. Forcomparison, we considered the standard EnKF up-dating scheme for non-linear likelihood models. Thekernelised shrinkage regression techniques providedgood estimates of unknown reference state vector, withuncertainty estimates close to the theoretical bounds.On this model, the updating scheme based on kernelPCR performed slightly better than kernel PLSR, al-though the subspace dimension tended to be smallerwhen using the supervised PLSR approach. The stan-dard EnKF, on the other hand, completely missed the

Comput Geosci (2011) 15:529–544 543

reference solution, with an ensemble collapsing after afew updating steps.

Acknowledgement This work is funded by the Uncertainty inReservoir Evaluation (URE) consortium at NTNU.

Open Access This article is distributed under the terms of theCreative Commons Attribution Noncommercial License whichpermits any noncommercial use, distribution, and reproductionin any medium, provided the original author(s) and source arecredited.

Appendix A Gaussian random field

The collection of random variables {r(x1), . . . , r(xne)} isa Gaussian random field (GRF) if any subset of therandom variables {r(xi)} has a joint Gaussian distrib-ution. A GRF is completely specified through a meanand covariance function, denoted m(x), and c(x, x′)respectively, for any vectors x and x′ ∈ R

nx×1. Here,m(x) = E[r(x)]] and

c(x, x′) = E[(r(x) − m(x))

(r(x′) − m(x′)

)T]

Whilst m : Rnx → R can be any function, the follow-

ing criteria must be satisfied for the covariance functionc : (Rnx×1 × R

nx×1) → R:

1. Symmetric: c(x, x′) = c(x′, x).2. Positive semidefinite:∫

c(x, x′) f (x) f (x′)dμ(x)dμ(x′) ≥ 0, (24)

for all x and x′ ∈ Rnx×1 and f ∈ L2(R

nx , μ).

Within the class of valid covariance functions are thestationary and isotropic:

c(x, x′) = c(‖x − x′‖2),

where ‖ · ‖2 is the Euclidean norm.

Appendix B Computational properties

If the model parameters, θ are specified, the com-putational complexity of the EnKF updating schemesbased on kernel RR, kernel PCR and kernel PLSR isO(max{nd, nx, ne}n2

e). Hence, we have the same compu-tational complexity and memory requirements as thefastest EnKF algorithms previously suggested whenne < max{nx, nd}. Further note that we can write the up-dating scheme based on kernel RR, similarly to kernelPCR, using the ne estimated eigenvectors and values ofthe centred kernel matrix. This follows because HCHby construction is symmetric, positive semidefinite.

When the model parameters, θ are selected basedon a CV scheme, as outlined in Algorithm 1, thecomputational complexity of the respective kernelisedEnKF updating schemes is O(r max{nx, nd, ne}n2

e). Inthe example considered in this paper, r = nσ ne, wherenσ denotes the number of discrete values of σ weconsider in the Gaussian kernel function, which in thisstudy is ne/4. Hence, in terms of the computationalcomplexity, it is preferable to pre-select the model pa-rameters because the computational demands decreaseby a factor n2

e . However, in the general case we expectthat using pre-selected model parameters will be aless robust choice, compared with the alternative ofselecting the model parameters at each updating stepusing CV.

References

1. Aanonsen, S.I., Nævdal, G., Oliver, D.S., Reynolds, A.C.,Vallès, B.: Ensemble Kalman filter in reservoir engineering—a review. SPE J. 14(3), 393–412 (2009)

2. Anderson, J.L.: A local least squares framework for ensemblefiltering. Mon. Weather Rev. 131(4), 634–642 (2003)

3. Caers, J., Park, K.: Distance-based representation of reser-voir uncertainty: the metric EnKF. In: Proceeding to theECMOR XI, Sept 8–11. Bergen, Norway (2008)

4. Casella, G., Berger, R.L.: Statistical Inference. Duxbury(2002)

5. Chiles, J.-P., Delfiner, P.: Geostatistics, Modeling Spatial un-certainty. Wiley, New York (1999)

6. Doucet, A., Godsill, S., Andrieu, C.: On sequential MonteCarlo sampling methods for Bayesian filtering. Stat. Comput.10(3), 197–208 (2000)

7. Evensen, G.: Sequential data assimilation with nonlinearquasi-geostrophic model using Monte Carlo methods to fore-cast error statistics. J. Geophys. Res. 99, 10143–10162 (1994)

8. Evensen, G.: Data Assimilation. The ensemble Kalman filter.Springer, New York (2007)

9. Farrer, D.E., Glauber, R.R.: Multicollinearity in regressionanalysis: the problem revisited. Rev. Econ. Stat. 49(1), 92–107(1967)

10. Furrer, R., Bengtsson, T.: Estimation of high-dimensionalprior and posterior covariance matrices in Kalman filter vari-ants. J Multivariate Anal. 98(2), 227–255 (2007)

11. Gibbs, M.N.: Bayesian Gaussian Processes for Regressionand Classification. Ph.D. thesis, University of Cambridge(1997)

12. Hastie, T., Tibshirani, R., Freidman, J.: The Elements of Sta-tistical Learning; Data Mining, Inference, and Prediction, 2ndedn. Springer, New York (2009)

13. Helland, I.S.: Some theoretical aspects of partial least squaresregression. Chemometr. Intell. Lab. Syst. 58(2), 97–107(2001)

14. Hoerl, A.E., Kennard, R.W.: Ridge Regression: Biased es-timation for nonorthogonal problems. Technometrics 12(3),55–67 (1970)

15. Hofmann, T., Schölkopf, B., Smola, A.J.: Kernel methods inmachine learning. Ann. Stat. 36(3), 1171–1220 (2008)

544 Comput Geosci (2011) 15:529–544

16. Hotelling, H.: Analysis of a complex of statistical variablesinto Principal Components. J. Educ. Psychol. 24(6), 417–441and 498–520 (1933)

17. Houtekamer, P.L., Mitchell, H.L.: Data assimilation using anensemble Kalman filter technique. Mon. Weather Rev. 126,796–811 (1998)

18. Huang, T.M., Kecman, V., Kopriva, I.: Kernel based algo-rithms for mining huge data sets, supervised, semi-supervised,and unsupervised learning. Springer, New York (2006)

19. Journel, A.G., Huijbregts, C.J.: Mining Geostatistics.Academic, London (1978)

20. Kalivas, J.H.: Cyclic subspace regression with analysis ofthe hat matrix. Chemometr. Intell. Lab. Syst. 45, 215–224(1999)

21. Kalman, R.E.: A new approach to linear filtering and predic-tion problems. Trans ASME J Basic Eng 82(Series D), 35–45(1960)

22. Kennedy, J., Eberhart, R.: Particle swarm optimization. In:Proceedings of the IEEE International Conference on NeuralNetworks, Piscataway, NJ, pp. 1942–1948 (1995)

23. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization bysimulated annealing Science 220(4598), 671–680 (1983)

24. Kwok, J.T., Tsang, I.W.: The pre-image problem in ker-nel methods. IEEE Trans. Neural Netw. 15(6), 1517–1525(2004)

25. Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate analysis.Academic, London (1979)

26. Mika, S., Schölkopf, B., Smola, A., Müller, R., Scholz, M.,Rätch, G.: Advances in neural information processing sys-tems 11, chapter kernel PCA and de-noising in feature spaces.Morgan Kaufmann, San Mateo (1998)

27. Myrseth, I.B., Omre, H.: Large-scale inverse problems andquantification of uncertainty, chapter the ensemble kalmanfilter and related filters. John Wiley and Sons (2010)

28. Myrseth, I.B., Sætrom, J., Omre, H.: Resampling the ensem-ble Kalman filter. Paper submitted for publication (2010)

29. Nocedal, J., Wright, S.: Numerical optimization. Springer,New York (2006)

30. O’Hagan, A.: Curve fitting and optimal design for prediction.J. R. Stat. Soc. B (Met.) 40(1), 1–42 (1978)

31. Oliver, D.: On conditional simulation to inaccurate data.Journal of Mathematical Geology 28(6), 811–817 (1996)

32. Rao, C.R.: Linear Statistical Inference and Its Applications,2nd edn. Wiley, New York (1973)

33. Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes forMachine Learning. MIT Press (2006)

34. Ränner, S., Lindgren, F., Gelandi, P., Wold, S.: A PLS kernelalgorithm for data sets with many variables and fewer objects.Part 1: Theory and Algorithm. J. Chemometr. 8(2), 111–125(1994)

35. Rosipal, R., Girolami, M., Trejo, L.J., Cichocki, A.: KernelPCA for feature extraction and de-noising in non-Linear re-gression. Neural Comput Appl 10(3), 231–243 (2001)

36. Rosipal, R., Trejo, L.J.: Kernel Partial Least Squares regres-sion in reproducing kernel Hilbert space. J Mach Learn Res2, 97–123 (2002)

37. Sacher, W., Bartello, P.: Sampling errors in EnsembleKalman filtering. Part I: Theory. Mon. Weather Rev. 136(8),3035–3049 (2008)

38. Sarma, P., Chen, W.H.: Generalization of the EnsembleKalman Filter using kernels for nongaussian Random Fields.In: Proceedings to the Reservoir Simulation Symposium,SPE, The Woodlands, Texas, 2–4 February 2009 (2009)

39. Sarma, P., Durlofsky, L., Aziz, K.: Kernel principal compo-nent analysis for an efficient, differentiable parametrizationof multipoint geostatistics. Math. Geol. 40(1), 3–32 (2008)

40. Scales, J.A., Snieder, R.: To Bayes or not to Bayes?Geophysics 62(4), 1045–1046 (1997)

41. Schölkopf, B., Smola, A., Muller, K.: Nonlinear componentanalysis as a kernel eigenvalue problem. Neural Comput.10(5), 1299–1319 (1998)

42. Seber, G.A.F., Lee, A.J.: Linear Regression Analysis. Wiley,New York (2003)

43. Smola, A.: Learning with Kernels. Ph.D. thesis, TU Berlin(1998)

44. Smola, A., Schölkopf, B.: Learning with Kernels. Springer,New York (2002)

45. Smola, A., Schölkopf, B., Müller, K.: The connection betweenregularization operators and support vector kernels. NeuralNetworks 11(4), 637–649 (1998)

46. Sætrom, J., Omre, H.: Ensemble Kalman filtering withshrinkage regression techniques. Comput. Geosci. 15(2), 271–292 (2011)

47. Taylor, J.S., Cristianini, N.: Kernel methods for pattern analy-sis. Cambridge University Press, Cambridge (2004)

48. van Leeuwen, P.J.: Comments on “Data assimilation using anensemble Kalman filter technique”. Mon. Weather Rev. 127,1374–1377 (1999)

49. Vapnik, V.: Statistical Learning Theory. Wiley, New York(1998)

50. Williams, C.K.I.: Learning and inference in graphical mod-els, chapter prediction with Gaussian processes: from linearregression to linear prediction and beyond. Kluwer, Norwell(1998)

51. Wold, H.: Quantitative sociology: international perspectiveson mathematical and statistical model building, chapter pathmodels with latent variables: the NiPALS Approach. pp. 307–357. Academic, London (1975)

52. Young, N.: An introduction to Hilbert Space. CambridgeUniversity Press, Cambridge (1988)