Developing fast low-rank tensor methods for solving PDEs with uncertain coefficients

Low-rank tensors for PDEs withuncertain coefficients

Alexander Litvinenko

Center for UncertaintyQuantification


Center for Uncertainty Quantification Logo Lock-up

http://sri-uq.kaust.edu.sa/

Extreme Computing Research Center, KAUST

Alexander Litvinenko Low-rank tensors for PDEs with uncertain coefficients

http://sri-uq.kaust.edu.sa/

4*

The structure of the talk

Part I (Stochastic Forward Problem):1. Motivation2. Elliptic PDE with uncertain coefficients3. Discretization and low-rank tensor approximations

Part II (Stochastic Inverse Problem via Bayesian Update):1. Bayesian update surrogate2. Examples

Part III (Different Examples relevant for BGS)

13

13

17

17

14

14 17

13

17

14 15

13 13

17 29

13 48

15

13 13

13 13

15 13

13

13 16

23

8 8

13 15

28 29

8

8 15

8 15

8 15

19

18 18

61

57

23

17 17

17 17

23 35

57 60

61 117

17

17 17

17 17

14 14

14

7 7

14 14

34

21 14

17 14

28 28

10

10 13

17 17

17 17

11 11

17

11 11

69

40

17 11

17 11

36 28

69 68

10

10 11

9 9

10 11

9

9 12

14 14

21 21

14

14

11

11 11

42

14

11 11

11 11

14 22

38 36

12

12 13

12 12

10 10

12

10 10

23

12 10

10 10

15 15

13

10 10

15 15

69

97

49

28

16 15

12 12

21 21

48 48

83 132

48 91

16

12 12

13 12

8 8

13

8 8

26

13 8

13 8

22 21

13

13 13

9 9

13 13

9

9 13

49

26

9 12

9 13

26 22

49 48

12

12 14

12 14

12 14

15

9 9

18 18

26

15 15

14 14

26 35

15

14 14

15 14

15 14

16

16 19

97

68

29

16 18

16 18

29 35

65 64

97 132

18

18 18

15 15

18 18

15

15

14

7 7

33

15 16

15 17

32 32

16

16 17

14 14

16 17

14

14 18

64

33

11 11

14 18

31 31

72 65

11

11

8

8 14

11 18

11 13

18

13 13

33

18 13

15 13

33 31

20

15 15

19 15

18 15

19

18 18

53

87

136

64

35

19 18

14 14

35 35

64 66

82 128

61 90

33 62

8

8 13

14 14

17 14

18 14

17

17 18

2917 18

10 10

35 35

19

10 10

13 10

19 10

13

13

10

10 14

70

28

13 15

13 13

29 37

56 56

15

13 13

15 13

15 13

19

19

10

10 15

23

11 11

12 12

28 33

11

11 12

11 12

11 12

18

15 15

115

66

23

18 15

18 15

23 30

49 49

121 121

18

18 18

12 12

18 18

12

12 18

22

11 11

11 11

27 27

11

11 11

11 11

10 10

17

10 10

6222

17 10

17 10

21 21

59 49

13

10 10

18 18

10 10

11 11

10

10 11

27

10 11

10 11

32 21

12

12 15

12 13

12 15

13

13 19

88

115

62

27

13 19

13 14

27 32

62 59

115 121

61 90

10

10 11

14 14

21 14

12 12

14

10 10

12 12

29

14 12

15 12

35 35

14

14 15

11 11

14 15

11

11

8

8 16

69

29

11 18

11 23

28 28

62 62

18

18

8

8 15

15 15

13 13

15

13 13

29

15 13

13 13

33 28

16

13 13

16 13

15 13

18

15 15

135

62

29

18 15

18 15

22 22

69 62

101 101

10

10 11

19 19

15 15

7 7

15

7 7

40

15 7

15 7

40 22

19

19

9

9 13

18 18

19 22

18

18

11

10 10

11 11

62

31

18 20

11 11

31 31

39 39

20

11 11

19 11

12 11

19

12 12

26

12 12

14 12

13 13

12

12 14

13 13




2

4*

My interests and collaborations




3

4*

Motivation to do Uncertainty Quantification (UQ)

Motivation: there is an urgent need to quantify and reduce theuncertainty in multiscale-multiphysics applications.

UQ and its relevance: Nowadays computational predictions areused in critical engineering decisions. But, how reliable arethese predictions?

Example: Saudi Aramco currently has a simulator,GigaPOWERS, which runs with 9 billion cells. How sensitiveare these simulations w.r.t. unknown reservoir properties?

My goal is systematic, mathematically founded, develop-ment of UQ methods and low-rank algorithms relevant forapplications.




4

4*

PDE with uncertain coefficient

Consider− div(κ(x , ω)∇u(x , ω)) = f (x , ω) in G × Ω, G ⊂ Rd ,u = 0 on ∂G,

where κ(x , ω) - uncertain diffusion coefficient.

1. Efficient Analysis of High Dimensional Data in TensorFormats, Espig, Hackbusch, Litvinenko., Matthies, Zander,2012.2. Efficient low-rank approx. of the stoch. Galerkin ma-trix in tensor formats, Wahnert, Espig, Hackbusch, A.L.,Matthies, 2013.3. PCE of random coefficients and the solution of stochas-tic PDEs in the Tensor Train format , Dolgov, Litvinenko,Khoromskij, Matthies, 2016.4. Application of H-matrices for computing the KL expan-sion, Khoromskij, Litvinenko, Matthies Computing 84 (1-2),49-67, 2009

0 0.5 1-20

0

20

40

60

50 realizations of the solution u,

the mean and quantiles

Related work by R. Scheichl, Chr. Schwab, A. Teckentrup, F. Nobile, D. Kressner,...




5

4*

Canonical and Tucker tensor formats

[Pictures are taken from B. Khoromskij and A. Auer lecture course]

Storage: O(nd )→ O(dRn) and O(Rd + dRn).




6

4*

Karhunen Loeve and Polynomial Chaos Expansions

Apply bothTruncated Karhunen Loeve Expansion (KLE):

κ(x , ω) ≈ κ0(x) +L∑

j=1

κjgj(x)ξj(θ(ω)),

where θ = θ(ω) = (θ1(ω), θ2(ω), ..., ),ξj(θ) = 1

κj

∫G (κ(x , ω)− κ0(x)) gj(x)dx .

Truncated Polynomial Chaos Expansion (PCE)

κ(x , ω) ≈∑

α∈JM,pκ(α)(x)Hα(θ)

ξj(θ) ≈∑

α∈JM,pξ(α)j Hα(θ).




7

4*

Discretization of elliptic PDE

Ku = f, where

K :=L∑`=1

K` ⊗M⊗µ=1

∆`µ, K` ∈ RN×N ,∆`µ ∈ RRµ×Rµ ,

u :=r∑

j=1

uj ⊗M⊗µ=1

ujµ, uj ∈ RN ,ujµ ∈ RRµ ,

f :=R∑

k=1

f k ⊗M⊗µ=1

gkµ, f k ∈ RN , gkµ ∈ RRµ .

Efficient low-rank approximation of the stochastic Galerkin matrix in tensor formats, Wahnert, Espig,Hackbusch, Litvinenko, Matthies, 2013.

In [2] we analyzed tensor ranks (compression properties) of thestochastic Galerkin operator K.




8

4*

Numerical Experiments

2D L-shape domain, N = 557 dofs.Total stochastic dimension is Mu = Mk + Mf = 20, there are|JM,p| = 231 PCE coefficients

u =231∑j=1

uj,0 ⊗20⊗µ=1

ujµ ∈ R557 ⊗20⊗µ=1

R3.

Tensor u has 320 · 557 ≈ 2 · 1012 entries ≈ 16 TB of memory.

Instead we store only 231 · (557 + 20 · 3) ≈ 144000 entries≈ 1.14 MB.




9

4*

Level sets

Now we compute level sets

ui : ui > b ·maxi

u,

i := (i1, ..., iM+1)for b ∈ 0.2, 0.4, 0.6, 0.8.

The computing time for each b was 10 minutes.




10

4*

Part II

Part II: Developing of cheap Bayesian updatesurrogate

1. Rosic, Litvinenko, Pajonk, Matthies J. Comp. Ph. 231 (17), 5761-5787, 20132. Inverse Problems in a Bayesian Setting, Matthies, Zander, Pajonk, Rosic, Litvinenko. Comp. Meth.forSolids and Fluids Multiscale Analysis, 2016

Related work by A. Stuart, Chr. Schwab, A. El Sheikh, Y.Marzouk, H. Najm, O. Ernst

4*

Numerical computation of Bayesian Update surrogate

Notation: y – measurements from engineers, y(ξ) – forecastfrom the simulator, ε(ω) – the Gaussian noise.

Look for ϕ such that q(ξ) = ϕ(z(ξ)), z(ξ) = y(ξ) + ε(ω):

ϕ ≈ ϕ =∑α∈Jp

ϕαΦα(z(ξ))

and minimize ‖q(ξ)− ϕ(z(ξ))‖2L2, where Φα are known

polynomials (e.g. Hermite).Taking derivatives with respect to ϕα:

∂

∂ϕα〈q(ξ)− ϕ(z(ξ)),q(ξ)− ϕ(z(ξ))〉 = 0 ∀α ∈ Jp




11

4*

Numerical computation of NLBU

∂

∂ϕαE

q2(ξ)− 2∑β∈J

qϕβΦβ(z) +∑β,γ∈J

ϕβϕγΦβ(z)Φγ(z)

= 2E

−qΦα(z) +∑β∈J

ϕβΦβ(z)Φα(z)

= 2

∑β∈J

E [Φβ(z)Φα(z)]ϕβ − E [qΦα(z)]

= 0 ∀α ∈ J .




12

4*


Now, rewriting the last sum in a matrix form, obtain the linearsystem of equations (=: A) to compute coefficients ϕβ: ... ... ...

... E [Φα(z(ξ))Φβ(z(ξ))]...

... ... ...

...ϕβ

...

=

...

E [q(ξ)Φα(z(ξ))]...

,

where α, β ∈ J , A is of size |J | × |J |.




13

4*


Finally, the assimilated parameter qa will be

qa = qf + ϕ(y)− ϕ(z), (1)

z(ξ) = y(ξ) + ε(ω),ϕ =

∑β∈Jp

ϕβΦβ(z(ξ))




14

4*

Example: 1D elliptic PDE with uncertain coeffs

−∇ · (κ(x , ξ)∇u(x , ξ)) = f (x , ξ), x ∈ [0,1]

+ Dirichlet random b.c. g(0, ξ) and g(1, ξ).3 measurements: u(0.3) = 22, s.d. 0.2, x(0.5) = 28, s.d. 0.3,x(0.8) = 18, s.d. 0.3.

κ(x, ξ): N = 100 dofs, M = 5, number of KLE terms 35, beta distribution for κ, Gaussian covκ, cov.length 0.1, multi-variate Hermite polynomial of order pκ = 2;

RHS f (x, ξ): Mf = 5, number of KLE terms 40, beta distribution for κ, exponential covf , cov. length 0.03,multi-variate Hermite polynomial of order pf = 2;

b.c. g(x, ξ): Mg = 2, number of KLE terms 2, normal distribution for g, Gaussian covg , cov. length 10,multi-variate Hermite polynomial of order pg = 1;

pφ = 3 and pu = 3




15

4*

Example: Updating of the parameter

0 0.5 10

0.5

1

1.5

0 0.5 10

0.5

1

1.5

Figure: Prior and posterior (updated) parameter κ.

Collaboration with Y. Marzouk, MIT, and TU Braunschweig.Together with H. Najm, Sandia Lab, we try to compare ourtechnique with his advanced MCMC technique for chemicalcombustion eqn.




16

4*

Example: updating of the solution u

0 0.5 1-20

0

20

40

60

0 0.5 1-20

0

20

40

60

0 0.5 1-20

0

20

40

60

0 0.5 1-20

0

20

40

60

0 0.5 1-20

0

20

40

60

Figure: Original and updated solutions, mean value plus/minus 1,2,3standard deviations. Number of available measurements 0,1,2,3,5

[graphics are built in the stochastic Galerkin library sglib, written by E. Zander in TU Braunschweig]




17

4*

Part III: My contribution to MUNA project

MUNA=Management and minimization of Uncertainties inNumerical Aerodynamics.

1. Quantification of airfoil geometry-induced aerodynamicuncertainties-comparison of approaches, Liu, Litvinenko,Schillings, Schulz, JUQ 2017

2. Numerical Methods for Uncertainty Quantification andBayesian Update in Aerodynamics Litvinenko, Matthies,chapter in Springer book, Vol 122, pp 262-285, 2013.

4*

Example: uncertainties in free stream turbulence

α

v

v

u

u’

α’

v1

2

Random vectors v1(θ) and v2(θ) model free stream turbulence




19

4*

Example: 3sigma intervals

Figure: 3σ interval, σ standard deviation, in each point of RAE2822airfoil for the pressure (cp) and friction (cf) coefficients.




20

4*

Mean and variance of density, tke, xv, zv, pressure




21

4*

Example: Kriging and geostat. optimal design

Domain: 20m × 20m × 20m, 25,000× 25,000× 25,000 dofs,4000 measurements.

Log-Permeability. Color scale: showing the 95% confidence interval.

Kriging and spatial design accelerated by orders of magnitude:Combining low-rank with FFT , Nowak, Litvinenko, Mathemati-cal Geosciences 45 (4), 411-435, 2013




22

4*

Numerics on computer with 16GB RAM:

1. 2D Kriging with 270 million estimation points and 100measurement values (0.25 sec.),

2. to compute the estimation variance (< 1 sec.),3. to evaluate the spatial average of the estimation variance

(the A-criterion of geostat. optimal design) for 2 · 1012

estim. points (30 sec.),4. to compute the C-criterion of geostat. optimal design for

2 · 1015 estim. points (30 sec.),5. solve 3D Kriging problem with 15 · 1012 estim. points and

4000 measurement data values (20 sec.)

Collaboration with Stuttgart University, hydrology andgeosciences.




23

4*

Example from spatial statistics

Goal: To improve estimation of un-known statistical parameters in a spa-tial soil moisture field, Mississippibasin, [−85 − 73]× [32,43].

Log-likelihood function with C = e−|x−y|θ and Z available

(satellite) data:

L(θ) = −n2

log(2π)− 12

log |C(θ)| − 12

Z>C(θ)−1Z .

Collaboration with statisticians: M. Genton, Y. Sun, R. Huser, H. Rue from KAUST.

n = 512K , matrix setup 261 sec., compression rate 99.98% (0.4 GB against 2006 GB).H-LU is done in

843 sec., error 2 · 10−3.




24

4*

Conclusion

Introduced:1. Low-rank tensor methods to solve PDEs with uncertain

coefficients,2. Post-processing in low-rank tensor format, computing level

sets3. Bayesian update surrogate ϕ (as a linear, quadratic,...

approximation)4. Quantification of uncertainties in Numerical Aerodynamics5. Applications in geosciences6. Estimating unknown coefficients in spatial statistics

(moisture example)




25

4*

Thank you

Thank you!




26

4*

Possible collaboration

1. Dominic Breit (error estimates for UQ applications tobalance statistical and discretization errors)

2. Gabriel Lord (num. meth. for PDEs with uncertainties;combination of multiscale methods, UQ techniques andBayesian inference for reservoir modeling; low-rank tensormethods for high-dimensional problems).

3. Lehel Banjai (computation of electromagnetic fieldsscattered from dielectric objects of uncertain shapes;balancing of the Runge-Kutta time discretization step andthe H-matrix approximation rank in time-dependent PDEs),




27

4*

Possible collaboration

1. BGS (CO2 storage, reservoir modeling, spatial statistics ingeology/geophysics),

2. Lyell Institute (subsurface flow under uncertainties, EOR,Bayesian techniques for data assimilations)

3. EGIS:3.1. Mike Christie (reservoir modeling under uncertainties,

EOR, seismic wave propagation in uncertain media)3.2. Vasily Demyanov (uncertainty quantification and

low-rank approximations in geostatistics)3.3. Dan Arnold (modeling of random geology, multi-scale,

Bayesian inference)3.4. Ahmed El Sheikh (fast Bayesian update methods,

advanced UQ, surrogates for BU, big data from spatialstatistics).




28

4*

My experience since 2002




29

4*

Explanation of Bayesian Update surrogate

Let the stochastic model of the measurement is given by

y =M(q) + ε, ε -measurement noise

Best estimator ϕ for q given z, i.e.

ϕ = argminϕ E[‖q(·)− ϕ(z(·))‖22].

The best estimate (or predictor) of q given themeasurement model is

qM(ξ) = ϕ(z(ξ))).

The remainder, i.e. the difference between q and qM, isgiven by

q⊥M(ξ) = q(ξ)− qM(ξ),

Due to the minimisation property of the MMSEestimator—orthogonal to qM(ξ), i.e. cov(q⊥M,qM) = 0.

[Thanks to Elmar Zander, TU Braunschweig]




30

In other words,

q(ξ) = qM(ξ) + q⊥M(ξ) (2)

yields an orthogonal decomposition of q.Actual measurement y , prediction q = ϕ(y). Part qM of qcan be “collapsed” to q. Updated stochastic model q′ isthus given by

q′(ξ) = q + q⊥M(ξ) (3)

q′(ξ) = q(ξ) + (ϕ(y)− ϕ(z(ξ))). (4)




31

4*

Future plans, Idea N1

Possible collaboration work 1 To develop a low-rank adaptivegoal-oriented Bayesian update technique. The solution of the forwardand inverse problems will be considered as a whole adaptiveprocess, controlled by error/uncertainty estimators.

z

(y - z) q

f ε

forward update

low-rank and adaptive

y

f z

(y - z)

ε

forwardy q.....

low-rank and adaptive

... q update

Stochastic forward spatial discret.

stochastic discret.

low-rank approx.

Inverse problem

Errors

inverse operator approx.

4*


Edge between Green functions in PDEs and covariancematrices.Possible collaboration with statistical group, Doug Nychka(NCAR), Haavard Rue




32

4*


Data assimilation techniques, Bayesian update surrogate.Develop non-linear, non-Gaussian Bayesian updateapproximation for gPCE coefficients.Possible collaboration with Kody Law (OAK NL), Y. Marzouk(MIT), H. Najm (Sandia NL), TU Braunschweig and KAUST.

4*

Example: Canonical rank d , whereas TT rank 2

d-Laplacian over uniform tensor grid. It is known to have theKronecker rank-d representation,

∆d = A⊗IN⊗...⊗IN +IN⊗A⊗...⊗IN +...+IN⊗IN⊗...⊗A ∈ RI⊗d⊗I⊗d

(5)with A = ∆1 = tridiag−1,2,−1 ∈ RN×N , and IN being theN × N identity. Notice that for the canonical rank we have rankkC(∆d ) = d , while TT-rank of ∆d is equal to 2 for anydimension due to the explicit representation

∆d = (∆1 I)×(

I 0∆1 I

)× ...×

(I 0

∆1 I

)×(

I∆1

)(6)

where the rank product operation ”×” is defined as a regularmatrix product of the two corresponding core matrices, theirblocks being multiplied by means of tensor product. The similarbound is true for the Tucker rank rankTuck (∆d ) = 2.

4*

Advantages and disadvantages

Denote k - rank, d-dimension, n = # dofs in 1D:

1. CP: ill-posed approx. alg-m, O(dnk), hard to computeapprox.

2. Tucker: reliable arithmetic based on SVD, O(dnk + kd )

3. Hierarchical Tucker: based on SVD, storage O(dnk + dk3),truncation O(dnk2 + dk4)

4. TT: based on SVD, O(dnk2) or O(dnk3), stable5. Quantics-TT: O(nd )→ O(d logqn)

4*

How to compute the variance in CP format

Let u ∈ Rr and

u := u − ud⊗µ=1

1nµ

1 =r+1∑j=1

d⊗µ=1

ujµ ∈ Rr+1, (7)

then the variance var(u) of u can be computed as follows

var(u) =〈u, u〉∏dµ=1 nµ

=1∏d

µ=1 nµ

⟨r+1∑i=1

d⊗µ=1

uiµ

,

r+1∑j=1

d⊗ν=1

ujν

⟩

=r+1∑i=1

r+1∑j=1

d∏µ=1

1nµ

⟨uiµ, ujµ

⟩.

Numerical cost is O(

(r + 1)2 ·∑d

µ=1 nµ)

.

4*

Computing QoI in low-rank tensor format

Now, we consider how tofind ‘level sets’,

for instance, all entries of tensor u from interval [a,b].

4*

Definitions of characteristic and sign functions

1. To compute level sets and frequencies we needcharacteristic function.2. To compute characteristic function we need sign function.

The characteristic χI(u) ∈ T of u ∈ T in I ⊂ R is for every multi-index i ∈ I pointwise defined as

(χI(u))i :=

1, ui ∈ I,0, ui /∈ I.

Furthermore, the sign(u) ∈ T is for all i ∈ I pointwise definedby

(sign(u))i :=

1, ui > 0;−1, ui < 0;0, ui = 0.




34

4*

sign(u) is needed for computing χI(u)

LemmaLet u ∈ T , a,b ∈ R, and 1 =

⊗dµ=1 1µ, where

1µ := (1, . . . ,1)t ∈ Rnµ .(i) If I = R<b, then we have χI(u) = 1

2(1+ sign(b1− u)).

(ii) If I = R>a, then we have χI(u) = 12(1− sign(a1− u)).

(iii) If I = (a,b), then we haveχI(u) = 1

2(sign(b1− u)− sign(a1− u)).

Computing sign(u), u ∈ Rr , via hybrid Newton-Schulz iterationwith rank truncation after each iteration.




35

4*

Level Set, Frequency

Definition (Level Set, Frequency)Let I ⊂ R and u ∈ T . The level set LI(u) ∈ T of u respect to I ispointwise defined by

(LI(u))i :=

ui ,ui ∈ I ;0,ui /∈ I ,

for all i ∈ I.The frequency FI(u) ∈ N of u respect to I is defined as

FI(u) := # suppχI(u).




36

4*

Computation of level sets and frequency

PropositionLet I ⊂ R, u ∈ T , and χI(u) its characteristic. We have

LI(u) = χI(u) u

and rank(LI(u)) ≤ rank(χI(u)) rank(u).The frequency FI(u) ∈ N of u respect to I is

FI(u) = 〈χI(u),1〉 ,

where 1 =⊗d

µ=1 1µ, 1µ := (1, . . . ,1)T ∈ Rnµ .




37

4*

Definition of tensor of order d

Tensor of order d is a multidimensional array over a d-tupleindex set I = I1 × · · · × Id ,

A = [ai1...id : i` ∈ I`] ∈ RI , I` = 1, ...,n`, ` = 1, ..,d .

A is an element of the linear space

Vn =d⊗`=1

V`, V` = RI`

equipped with the Euclidean scalar product 〈·, ·〉 : Vn ×Vn → R,defined as

〈A,B〉 :=∑

(i1...id )∈I

ai1...id bi1...id , for A, B ∈ Vn.




38

4*

Examples of rank-1 and rank-2 tensors

Rank-1:f (x1, ..., xd ) = exp(f1(x1) + ...+ fd (xd )) =

∏dj=1 exp(fj(xj))

Rank-2: f (x1, ..., xd ) = sin(∑d

j=1 xj), since

2i · sin(∑d

j=1 xj) = ei∑d

j=1 xj − e−i∑d

j=1 xj

Rank-d function f (x1, ..., xd ) = x1 + x2 + ...+ xd can beapproximated by rank-2: with any prescribed accuracy:

f ≈∏d

j=1(1 + εxj)

ε−∏d

j=1 1ε

+O(ε), as ε→ 0




39

4*

Conditional probability and expectation

Classically, Bayes’s theorem gives conditional probability

P(Iq|Mz) =P(Mz |Iq)

P(Mz)P(Iq) (orπq(q|z) =

p(z|q)

Zspq(q));

Expectation with this posterior measure is conditionalexpectation.

Kolmogorov starts from conditional expectation E (·|Mz),from this conditional probability via P(Iq|Mz) = E

(χIq |Mz

).




40

4*

Conditional expectation

The conditional expectation is defined asorthogonal projection onto the closed subspace L2(Ω,P, σ(z)):

E(q|σ(z)) := PQ∞q = argminq∈L2(Ω,P,σ(z)) ‖q − q‖2L2

The subspace Q∞ := L2(Ω,P, σ(z)) represents the availableinformation.

The update, also called the assimilated valueqa(ω) := PQ∞q = E(q|σ(z)), is a Q-valued RV

and represents new state of knowledge after the measurement.Doob-Dynkin: Q∞ = ϕ ∈ Q : ϕ = φ z, φmeasurable.




41

Engineering

Developing fast low-rank tensor methods for solving PDEs with uncertain coefficients