Introduction Introduction UQ Software PCES References

Polynomial Chaos based uncertainty propagationLecture 1: Uncertainty and Spectral Expansions

Bert Debusschere†, Cosmin Safta, Khachik Sargsyan, Habib Najm

†bjdebus@sandia.gov

Sandia National Laboratories,Livermore, CA

KUL UQ Summer School

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Acknowledgement

Omar Knio — Duke University, Raleigh, NCRoger Ghanem — University of Southern California, Los Angeles, CAOlivier Le Maître — LIMSI-CNRS, Orsay, Franceand many others ...

This work was supported by:

• US Department of Energy (DOE), Office of Advanced Scientific Computing Research (ASCR), ScientificDiscovery through Advanced Computing (SciDAC) and Applied Mathematics Research (AMR) programs.

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary ofLockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contractDE-AC04-94AL85000.

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Goals of this tutorial

• An overview of the key aspects of uncertainty quantification• 4 Lectures

• Lecture 1: Context and Fundamentals• Introduction to uncertainty• The various aspects of UQ• Software for UQ• Spectral representation of random variables

• Lecture 2: Forward Propagation• Lecture 3: Characterization• Lecture 4: Advanced Topics

• Please do not hesitate to ask questions!!

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Outline

1 Introduction

2 The Many Aspects of Uncertainty Quantification

3 Software for Uncertainty Quantification

4 Spectral Representation of Random Variables

5 References

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Aspects of Uncertainty Quantification

• Enabling Predictive Simulation• Types of uncertainty

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Predictive simulations enable science-based design

• Emperical design is inefficient and costly• Trial and error does not work well for complex systems

• Experiments not always feasible or permissible• Reliability of nuclear weapons• Climate change mitigation approaches

• Predictive simulations provide insight into the underlying physicsthat drive complex systems

• Identification of key mechanisms• Allows for rigorous optimization strategies

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Model validation requires targeted experimentsModel validation requires targeted experiments

DLR-A Flame: Red = 15,200!Fuel: 22.1% CH4, 33.2% H2, 44.7% N2!

Coflow: 99.2% Air, 0.8% H2O!Detailed Chemistry and Transport: 12-Step

Mechanism (J.-Y. Chen, UC Berkeley)!

Experiment!

LES!

x/d = 10!

MEAN!RMS!

Temperature!

Mixture Fraction!

Comparisons with 1D Raman/Rayleigh/CO-LIF

line images (Barlow et al. )!

H2O Mass Fraction!

CO Mass Fraction!

Large Eddy Simulation (LES) validation on turbulent flame (courtesy J.Oefelein, SNL)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Predictive simulation requires careful assessment of all sources of error anduncertainty

• Numerical errors• Grid resolution• Time step• Time integration order• Spatial derivative order• Tolerance on iterative solvers• Condition number of matrices

• Uncertainties• Initial and boundary conditions• Model parameters• Model equations• Coupling between models• Sampling noise in particle models• Stochastic forcing terms

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Governing equations for LES simulation of multiphase reacting flow

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Constitutive models provide closure for governing equations

Smagorinsky sub-grid scale model

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Dynamic modeling and reacting flows involve additional complexity

Does additional complexity provide more accuracy and less uncertainty?Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Experimental data is used to calibrate models

Predictive Simulation

Theory

Experiments

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

UQ methods extract information from all sources to enable predictivesimulation

Predictive Simulation

Theory

Experiments

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)p(λ, I)

P(D)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)p(λ, I)

P(D)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)p(λ, I)

P(D)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)p(λ, I)

P(D)

Debusschere – SNL UQ

Inference

Forward Propagation

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)P(λ, I)

P(D)

λ u

P(u) P(λ)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)P(λ, I)

P(D)

λ u

P(u) P(λ)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)P(λ, I)

P(D)

λ u

P(u) P(λ)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)P(λ, I)

P(D)

λ u

P(u) P(λ)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES Summary References

Formulation

dudt

= f (u;λ)

P(λ|D, I) =P(D|λ, I)P(λ, I)

P(D)

λ u

P(u) P(λ)

Debusschere – SNL UQ

• UQ not just about propagating uncertainties• The term UQ covers a wide range of methods

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

UQ assesses confidence in model predictions and allows resource allocationfor fidelity improvements

• Parameter inference• Determine parameters from data• Characterize uncertainties in inferred parameters

• Propagate input uncertainties through computational models• Account for uncertainty from all sources• Resolve coupling between sources

• Analysis• Sensitivity analysis• Attributions

• Model calibration, validation, selection, averaging

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Types of uncertainty

• Epistemic uncertainty• Variable has one particular value, but we don’t know what it is• Reducible: by taking more measurements, we can get to know the value of

the variable better• Examples

• The mass of the planet Neptune• Temperature of the ocean at a particular point and time

• Aleatory uncertainty• Intrinsic or inherent uncertainty: variable is random; different value each time

it is observed• Irreducible: taking more measurements will not reduce uncertainty in the

value of the variable• Examples:

• Collisions interactions in molecular systems• Sampling noise

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Viewpoints matter

• Epistemic versus aleatory sometimes depends on scale level• In fully resolved turbulent simulations, velocity field near a wall is

deterministic• In mesoscale channel flow, near wall velocity field is turbulent forcing term

(aleatory)• In macroscale channel flow, near wall velocity field provides friction term

(epistemic)

• These lectures follow the Bayesian view: probability represents thedegree of belief in the value of a variable

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Software for Uncertainty Quantification

• UQ in computational sciences is maturing• Algorithms and research codes are transitionting into engineering

sciences software• US DOE puts high importance on the role of UQ, especially as scientific

research moves into extreme scale• QUEST: Quantification of Uncertainty in Extreme Scale Simulations

• DOE SciDAC institute: http://quest-scidac.org/• One of the key goals is to make extreme scale UQ approaches available to

the scientific community• Various UQ methods offered under 4 software products

• DAKOTA• QUESO• GPMSA• UQTk

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

UQ Tools – DAKOTA: Design Analysis Kit for Optimization and TerascaleApplications

• http://dakota.sandia.gov/

• Large-Scale Optimization, non-intrusive UQ• Model calibration, global sensitivity analysis• Design of Experiments• solution verification, and parametric studies• Over 4000 download registrations

– govt, industry, academia• Generic interface to black-box application model• Java front end• Used in a wide variety of applications• GNU LPGL license

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

UQ Tools – QUESO

• MPI/C++ library• Large-scale inverse UQ• Statistical algorithms for Bayesian inference• model calibration, model validation,• decision making under uncertainty• Parallel multi-chain MCMC• Being integrated with DAKOTA

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

UQ Tools – UQTk: Uncertainty Quantification Toolkit

• http://www.sandia.gov/UQToolkit/

• Lightweight C++ library• Intrusive Polynomial Chaos UQ methods• Non-intrusive sparse-quadrature UQ• Bayesian inference• Custom representations of uncertain information• Well suited for prototyping, tutorial/training purposes

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

UQ Tools – GPMSA: Gaussian Process Models for Simulation Analysis

• Serial matlab code – C++ conversion plans• Integration with DAKOTA• Bayesian inference• Gaussian process surrogates• Global sensitivity analysis, forward UQ• Model calibration/parameter estimation• Statistical models to characterize model discrepancy or structural model

error• Prototyping tool

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Polynomial Chaos Expansions (PCEs)

• Representation of random variables with PCEs• Convergence• Galerkin projection• Basis types

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

One-Dimensional Hermite Polynomials

ψ0(x) = 1

ψk (x) = (−1)k ex2/2 dk

dxk e−x2/2, k = 1, 2, . . .

ψ1(x) = x , ψ2(x) = x2 − 1, ψ3(x) = x3 − 3x , . . .

The Hermite polynomials form an orthogonal basis over [−∞,∞] withrespect to the inner product

〈ψiψj〉 ≡1√2π

∫ ∞−∞

ψi (x)ψj (x)w(x)dx = δij

⟨ψ2

i

⟩where w(x) is the weight function

w(x) = e−x2/2

Note that e−x2/2√

2πis the density of a standard normal random variable

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

One Dimensional Polynomial Chaos Expansion

Consider:

u =P∑

k=0

ukψk (ξ)

• u: Random Variable (RV)represented with 1D PCE

• uk : PC coefficients (deterministic)• ψk : 1D Hermite polynomial of

order k• ξ: Gaussian RV

0.0 0.5 1.0 1.5 2.0u

0

1

2

3

4

5

Prob. Dens. [-]

u = 0.5 + 0.2ψ1(ξ) + 0.1ψ2(ξ)

A random quantity is represented with an expansion consisting of functions ofrandom variables multiplied with deterministic coefficients• Set of deterministic PC coefficients fully describes RV• Separates randomness from deterministic dimensions

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Multidimensional Hermite Polynomials

The multidimensional Hermite polynomial Ψi (ξ1, . . . , ξn) is a tensor product ofthe 1D Hermite polynomials, with a suitable multi-index αi = (αi

1, αi2, . . . , α

in),

Ψi (ξ1, . . . , ξn) =n∏

k=1

ψαik(ξk )

For example, 2D Hermite polynomials:

i p Ψi αi

0 0 1 (0,0)1 1 ξ1 (1,0)2 1 ξ2 (0,1)3 2 ξ2

1 − 1 (2,0)4 2 ξ1ξ2 (1,1)5 2 ξ2

2 − 1 (0,2)... ... ... ...

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Multidimensional Inner Products — Orthogonality

〈Ψi Ψj〉 ≡∫. . .

∫Ψi (ξ)Ψj (ξ)g(ξ1)g(ξ2) · · · g(ξn)dξ1dξ2 · · · dξn

=n∏

k=1

⟨ψαi

k(ξk )ψ

αjk(ξk )

⟩= δij

⟨Ψ2

i

⟩where, g(ξ) =

e−ξ2/2

√2π

such that,

u =P∑

k=0

uk Ψk ⇒ 〈Ψiu〉 =P∑

k=0

uk 〈Ψi Ψk 〉 = ui

⟨Ψ2

i

⟩

⇒ ui =〈uΨi〉⟨

Ψ2i

⟩

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Multidimensional Polynomial Chaos Expansion

Consider:

u =P∑

k=0

uk Ψk (ξ1, . . . , ξn)

• u: Random Variable (RV) represented with multi-D PCE• uk : PC coefficients (deterministic)• Ψk : Multi-D Hermite polynomials up to order p• ξi : Gaussian RV• n: Dimensionality of stochastic space• P + 1: Number of PC terms: P + 1 = (n+p)!

n!p!

The number of stochastic dimensions represents the number of independentinputs, degrees of freedom that affect the random variable u• E.g. one stochastic dimension per uncertain model parameter• Contributions from each uncertain input can be identified• Compact representation of a random variable and its dependencies

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Generalized Polynomial Chaos

PC Type Domain Density w(ξ) Polynomial Free parameters

Gauss-Hermite (−∞,+∞) 1√2π

e−ξ22 Hermite none

Legendre-Uniform [−1, 1] 12 Legendre none

Gamma-Laguerre [0,+∞) xαe−ξΓ(α+1)

Laguerre α > −1

Beta-Jacobi [−1, 1] (1+ξ)α(1−ξ)β

2α+β+1B(α+1,β+1)Jacobi α > −1, β > −1

Inner product: 〈ψiψj〉 ≡∫ b

a ψi (ξ)ψj (ξ)w(ξ)dξ

• Wiener-Askey scheme provides a hierarchy of possible continuous PCbases, see Xiu and Karniadakis, SISC, 2002.

• Legendre-Uniform PC is a special case of Beta-Jacobi PC• Beta-Jacobi PC allows tailored accuracy of the PC representation across the

domain

• Input parameter domain often dictates the most convenient choice of PC• Polynomials functions can also be tailored to be orthogonal w.r.t.

chosen, arbitrary density

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Gamma Distribution

0 2 4 6 8 10 12 14 16x

0.0

0.2

0.4

0.6

0.8

1.0

PD

F

α=0

α=1

α=2

α=3

α=4

• Useful to represent quantities that are strictly positive• Support on [0,+∞)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Beta Distribution

1.0 0.5 0.0 0.5 1.0x

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

PD

F

α=β=0.5

α=5,β=1

α=1,β=3

α=2,β=2

α=2,β=5

• Useful to represent quantities that vary between set boundaries• Can be tailored to give more importance to specific areas• Support on [−1, 1]

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

More Formally: Probability Spaces and Random Variables

• Let (Θ,S,P) be a probability space• Θ is an event space• S is a σ-algebra on Θ• P is a probability measure on (Θ,S)

• Random variables are functions X : Θ→ R with a measurecorresponding to their image:

• if X−1(A) ∈ S, then define µ(A) = P(X−1(A)).• p(x) = dµ/dx : the density of the random variable X (with respect to

Lebesgue measure on R).• Expectation: 〈f 〉 =

∫f dµ =

∫f p(x) dx

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Convergence of PC expansions

• General convergence theorems are subject of ongoing research• Depends on the type of underlying random variables ξ

• Wiener-Hermite Chaos has been well-studied• Generalized PC less so

• Ernst et al. 2011:• Let ξ : Θ→ RN such that for i = 1, . . . ,N each ξi : Θ→ R, be a set of

random variables• S(ξ): σ-algebra generated by the set ξ of random variables• L2(Θ,S(ξ),P): Hilbert space of real-valued random variables defined on

(Θ,S(ξ),P) with finite second moments• Any random variable in this σ-algebra can be represented with a Polynomial

Chaos expansion with germ ξ

• Does not say that any RV with finite variance can be represented with aPCE to arbitrary precision

• In practice, one rarely knows how many degrees of freedom a RV has• Also, RV specification is rarely complete• Often, in an engineering sense, PCE of a given order is seen as a model

choice to represent what is known about a RV

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

How do I know my PCE is converged?

• Approximation error in PCE is topic of a lot of research• Rules of thumb:

• Higher order PC coefficients should decay• Increase order until results no longer change• Not always fail-proof ...

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Postprocessing / Analysis

• Moments• Plotting PDFs of RVs represented with PCEs

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Moments of RVs described with PCEs

u =P∑

k=0

uk Ψk (ξ)

• Expectation: 〈u〉 = u0

• Variance σ2

σ2 =⟨

(u − 〈u〉)2⟩

=

⟨(

P∑k=1

uk Ψk (ξ))2

⟩

=

⟨P∑

k=1

P∑j=1

ujuk Ψj (ξ)Ψk (ξ)

⟩

=P∑

k=1

P∑j=1

ujuk 〈Ψj (ξ)Ψk (ξ)〉

=P∑

k=1

u2k

⟨Ψk (ξ)2

⟩Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Plotting PDFs of RVs corresponding to PCEs

u =P∑

k=0

uk Ψk (ξ)

• Analytical formula formula for PDF(u) exists• Involves polynomial root finding, and is hard to generalize to multi-D

• PCE is cheap to sample• Brute-force sampling and bin samples into histogram• Use Kernel Density Estimation (KDE) to get smoother PDF with fewer

samples ui

PDF(u) =1

Nsh

Ns∑i=1

K(

u − ui

h

)K is the kernel, h is the bandwidth.

• Gaussian KDE is commonly used with K (x) = 1√2π

e−x22 , leading to

PDF(u) =1

√2πNsh

Ns∑i=1

exp

(−

(u − ui )2

2h2

)

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Comparison of histograms and KDE

Source Wikipedia, Drleft; licensed under the Creative Commons Attribution-ShareAlike 3.0 License

• Bandwidth h needs to be chosen carefully to avoid oversmoothing

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

KDE requires judicious choice of bandwidth h

−5 −4 −3 −2 −1 0 1 2 3 4

Random variable

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

PD

F

True PDFw=wopt=0.16

w=2wopt

w=wopt/2

w=wopt/4

binning

• Bandwidth h needs to be chosen carefully to avoid oversmoothing

• Scott’s rule-of-thumb is optimal for Gaussian r. v., h = N−1

d+4

• With smaller h, the KDE becomes more accurate, but more noisy• Binning is similar to smaller h case, but it requires a good choice of Nbins

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

KDE requires judicious choice of bandwidth h

−6 −4 −2 0 2 4 6

Random variable

0.00

0.05

0.10

0.15

0.20

0.25

0.30

PD

F

True PDFw=wopt=0.16

w=2wopt

w=wopt/2

w=wopt/4

binning

• Bandwidth h needs to be chosen carefully to avoid oversmoothing

• Scott’s rule-of-thumb is optimal for Gaussian r. v., h = N−1

d+4

• With smaller h, the KDE becomes more accurate, but more noisy• Binning is similar to smaller h case, but it requires a good choice of Nbins

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

KDE requires judicious choice of bandwidth h

−10 −5 0 5 10

Random variable

0.00

0.05

0.10

0.15

0.20

0.25

0.30

PD

F

True PDFw=wopt=0.16

w=2wopt

w=wopt/2

w=wopt/4

binning

• Bandwidth h needs to be chosen carefully to avoid oversmoothing

• Scott’s rule-of-thumb is optimal for Gaussian r. v., h = N−1

d+4

• With smaller h, the KDE becomes more accurate, but more noisy• Binning is similar to smaller h case, but it requires a good choice of Nbins

Debusschere – SNL UQ

Introduction Introduction UQ Software PCES References

Further Reading

• N. Wiener, “Homogeneous Chaos", American Journal of Mathematics, 60:4, pp. 897-936, 1938.

• M. Rosenblatt, “Remarks on a Multivariate Transformation", Ann. Math. Statist., 23:3, pp. 470-472, 1952.

• R. Ghanem and P. Spanos, “Stochastic Finite Elements: a Spectral Approach", Springer, 1991.

• O. Le Maître and O. Knio, “Spectral Methods for Uncertainty Quantification with Applications to Computational Fluid Dynamics",Springer, 2010.

• D. Xiu, “Numerical Methods for Stochastic Computations: A Spectral Method Approach", Princeton U. Press, 2010.

• O.G. Ernst, A. Mugler, H.-J. Starkloff, and E. Ullmann, “On the convergence of generalized polynomial chaos expansions,” ESAIM:M2AN, 46:2, pp. 317-339, 2011.

• D. Xiu and G.E. Karniadakis, “The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations", SIAM J. Sci. Comput.,24:2, 2002.

• Le Maître, Ghanem, Knio, and Najm, J. Comp. Phys., 197:28-57 (2004)

• Le Maître, Najm, Ghanem, and Knio, J. Comp. Phys., 197:502-531 (2004)

• B. Debusschere, H. Najm, P. Pébay, O. Knio, R. Ghanem and O. Le Maître, “Numerical Challenges in the Use of Polynomial ChaosRepresentations for Stochastic Proecesses", SIAM J. Sci. Comp., 26:2, 2004.

• S. Ji, Y. Xue and L. Carin, “Bayesian Compressive Sensing", IEEE Trans. Signal Proc., 56:6, 2008.

• K. Sargsyan, B. Debusschere, H. Najm and O. Le Maître, “Spectral representation and reduced order modeling of the dynamics ofstochastic reaction networks via adaptive data partitioning". SIAM J. Sci. Comp., 31:6, 2010.

• K. Sargsyan, B. Debusschere, H. Najm and Y. Marzouk, “Bayesian inference of spectral expansions for predictability assessment instochastic reaction networks". J. Comp. Theor. Nanosc., 6:10, 2009.

Debusschere – SNL UQ