Large-Scale Inverse Problems

Large-scale Inverse Problems

Tania Bakhos, Peter KitanidisInstitute for Computational Mathematical Engineering, Stanford University

Arvind K. SaibabaDepartment of Electrical and Computer Engineering,Tufts University

June 28, 2015

Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 1 / 114

Outline

1 Introduction

2 Linear Inverse Problems

3 Geostatistical ApproachBayes’ theoremCoin toss exampleCovariance modelingNon-Gaussian priors

4 Data AssimilationApplication: CO2 monitoring

5 Uncertainty quantificationMCMC

6 Concluding remarks


What is an Inverse Problem?

Parameters s

Modelh(s)

Data y

Quantitiesof Interest

Inverse Problems



Parameters s

Modelh(s)

Data y


Inverse Problems



Parameters s

Modelh(s)

Data y


Inverse Problems


Inverse problems: Applications

Inverse Problems Geosciences

CO2

monitoringin the

subsurface

Contaminantsource iden-

tification

Climatechange

HydraulicTomog-raphy


Inverse problems: Applications

Inverse Problems Other fields

MedicalImaging

Non-destructive

testing

Neuroscience

ImageDeblurring


Application: Contaminant source identification1

1http://www.solinst.com/Prod/660/660d2.html, Stockie, SIAM Review 2011Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 5 / 114


Initialconditions

Transportprocesses

Predictions/Measurements



Initialconditions

Transportprocesses

Predictions/Measurements


Application: Hydraulic Tomography

Manage underground sites

To better locate naturalresources

Contaminant remediation

Source http://web.stanford.edu/ jonghyun/research.htmlBakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 6 / 114

Field pictures


Transient Hydraulic Tomography

Results from a field experiment conducted at the Boise Hydrological Research Site(BHRS) 2

Figure 1 : Hydraulic head measurements at observation wells (left) and log10 estimate ofthe hydraulic conductivity (right)

2Cardiff, Barrash and Kitanidis - Water Resoures Research 47(12) 2011.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 8 / 114

CSEM: Oil Exploration

Source: Morten et al, 72nd EAGE Conference 2010 Barcelona, and Newman et al.Geophysics, 72(2) 2010;


Monitoring CO2 emissions

Atmospheric transport model

Observations from monitoring stations, satellite observations, etc

Source: Anna Michalak’s plenary talkhttps://www.pathlms.com/siam/courses/1043/sections/1257


Application: Global Seismic Inversion

Bui-Thanh, Tan, et al. SISC 35.6 (2013): A2494-A2523.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 11 / 114

Need for Uncertainty Quantification

“ Uncertainty quantification (UQ) is the science of quantitative characterizationand reduction of uncertainties in applications. It tries to determine how likelycertain outcomes are if some aspects of the system are not exactly known.” -Wikipedia.

“ ... how do we quantify uncertainties in the predictions of our large-scalesimulations, given limitations in observational data, computational resources, andour understanding of physical processes ?”6

“ Well, what I’m saying is that there are known knowns and that there are knownunknowns. But there are also unknown unknowns; things we don’t know that wedon’t know. ”

- Gin Rummy, paraphrasing D. Rumsfeld.

6Bui et al. Proceedings of the International Conference on High Performance Computing,Networking, Storage and Analysis. IEEE Computer Society Press 2012
















Statistical framework for inverse problems

Estimate model parameters (and uncertainties) from data.

Propagate forward uncertainties to predict quantities and uncertainties.

Optimal experiment designI What experimental conditions yield the most information?

Challenge: framework often intractable because

Mathematically ill-posed (sensitivity to noise)

Computationally challenging problem

Insufficient information from data


Statistical framework for inverse problems

Estimate model parameters (and uncertainties) from data.

Propagate forward uncertainties to predict quantities and uncertainties.

Optimal experiment designI What experimental conditions yield the most information?

Challenge: framework often intractable because

Mathematically ill-posed (sensitivity to noise)

Computationally challenging problem

Insufficient information from data


Opportunities and challenges

Central question in our research

How to exploit structure in order to overcome the curse of dimensionality todevelop scalable algorithms for statistical inverse problems?

What do we mean by scalable?

amount of data

discretization of unknown random field

number of processors


Opportunities and challenges

Central question in our research

How to exploit structure in order to overcome the curse of dimensionality todevelop scalable algorithms for statistical inverse problems?

What do we mean by scalable?

amount of data

discretization of unknown random field

number of processors


Sessions at SIAM Geosciences

Plenary talks

IP1 The Seismic Inverse Problem Towards Wave Equation Based VelocityEstimation

I Fons ten Kroode, Shell Research, The NetherlandsI McCaw Hall 8:30-9:15 AM (Monday)

Contributed Talks

CP 3: Inverse ModelingI 4:30 PM - 6:30 PM, Monday June 29 th, Room: Fisher Conference Center

room #5


Minisymposia at SIAM GeosciencesMS 54 Recent advances in Geophysical Inverse Problems

I Tania Bakhos, Peter Kitanidis, Arvind SaibabaI 9:30 AM - 11:30 AM Thursday July 2, Room: Bechtel Conference Center -

Main Hall

MS 12 Bayesian Methods for Large-scale Geophysical Inverse ProblemsI Omar Ghattas, Noemi Petra, Georg StadlerI 2:00 PM - 4:00 PM, Monday June 29, Room: Fisher Conference Center room

#4

MS2, MS9, MS 15 Full-waveform inversionI William Symes, Hughes DjikpesseI 9:30 AM - 11:30 AM, 2:00 - 4:00 PM and 4:30 - 6:30 PMI Room: Fisher Conference Center room #1

MS 19 Full Waveform Inversion

MS 36 3D Elastic Waveform Inversion: Challenges in Modeling and Inversion

MS 58 Forward and Inverse Problems in Geodesy, Geodynamics, andGeomagnetism

MS46 Data Assimilation in Subsurface Applications: Advances in ModelUncertainty Quantification


Outline

1 Introduction







Introduction

What is an inverse problem?

Forward problem: Compute the output given a system and an input.Inverse problem: Compute either the input or the system given the output.

Hansen, PC. Discrete inverse problems: insight and algorithms. Vol. 7. SIAM, 2010Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 18 / 114

Example

Figure 2 : Magnetization inside volcano of Mt. Vesuvius from measurements of magneticfield

Hansen, Per Christian. Discrete inverse problems: insight and algorithms. Vol. 7. SIAM,2010


Challenges

Inverse problems are ill-posed. They do not satisfy the three conditions forwell-posedness.

Existence: The problem must have at least a solution.

Uniqueness: The problem must only have one solution.

Stability: The solution depends continuously on the data.

The mathematical term well-posed problem stems from a definition given byJacques Hadamard.


Image processing

Consider the equation,

y = Ax + ε

Notation:

b : observations - the blurry image.x : true image, we want to estimate.A : blurring operator - given.ε : noise in the data

Forward problem:Given the true image x and the blurring matrix A, we get the blurred image b.

What is the inverse problem?

The opposite of the forward problem. Given b and A, we compute x (the trueimage).


Image processing


y = Ax + ε





Image processing


y = Ax + ε





Image processing


y = Ax + ε





Image processing

From http://www.math.vt.edu/people/jmchung/resources/CSGF07.pdfBakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 22 / 114

Review of basic linear algebra

A square real matrix U ∈ Rn×n is orthogonal if its inverse equals itstranspose, i.e. UUT = I and UTU = I .

A real symmetric matrix A = AT has a spectral decomposition, A = UΛUT

where U is orthogonal and Λ = diag(λ1, ..., λn) is a diagonal matrix whoseentries are eigenvalues of A.

A real square matrix that is not symmetric can be diagonalized by twoorthogonal matrices with the singular value decomposition (SVD),A = UΣV T where Σ is a diagonal matrix whose entries are the singularvalues of A.


Need for regularizationPerturbation theory

Ax = b Would like to solve

A(x + δx) = b + ε Instead solving

Subtracting equation (2) - equation (1)

Aδx = ε ⇒ δx = A−1ε

Can show the following bounds

‖δx‖2 ≤ ‖A−1‖2‖ε‖2 ‖x‖2 ≥‖A‖2

‖b‖2

Important result

‖δx‖2

‖x‖2≤ ‖A‖2‖A−1‖2︸︷︷︸

cond(A)

‖ε‖2

‖b‖2

The more ill-conditioned the blurring operator A is, the worse is the reconstruction.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 24 / 114

TSVD

Regularization controls the amplification of noise.

Truncated SVD: Discard all the singular values that are smaller than a chosennumber.

The naive solution was given by

x = A−1b = VΣ−1UTb =N∑i=1

uTi b

σivi

For TSVD we truncate the singular values so the solution is given by,

xk =k∑

i=1

uTi b

σivi k < N

This yields the same solution as imposing a minimum 2-norm constraint on theleast squares problem minx‖Ax − b‖2.


TSVD


Truncated SVD: Discard all the singular values that are smaller than a chosennumber. The naive solution was given by


uTi b

σivi


xk =k∑

i=1

uTi b

σivi k < N



TSVD


Truncated SVD: Discard all the singular values that are smaller than a chosennumber. The naive solution was given by


uTi b

σivi


xk =k∑

i=1

uTi b

σivi k < N



TSVD

Figure 3 : Exact image (top left), TSVD k = 658 (top right), k = 218 (bottom left) andk = 7243 (bottom right)

658 was too low (over-smoothed) and 7243 too high (under-smoothed).

Hansen, PC. Discrete inverse problems: insight and algorithms. Vol. 7.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 26 / 114

Selective SVD

A variant of the TSVD is the SSVD where we only include components thatsignificantly contribute to the regularized solution. Given a threshold τ ,

x =∑|uT

i b|>τ

uTi b

σivi

This method is advantageous when some of the components uTi b correspondingto large singular values are small.


Tikhonov regularization

Least squares objective function

x = arg minx‖Ax − b‖2

2 + α2‖x‖22

where α is a regularization parameter.

The first term ‖Ax − b‖22 measures how well the solution predicts the noisy

data, sometimes referred to as “goodness-of-fit”.

The second term ‖x‖22 measures the regularity of the solution.

The balance of the terms is controlled by the parameter α.


Relation between Tikhonov and TSVDThe solution to the Tikhonov problem is given by,

xα = (ATA + α2I )−1ATb

If we replace A by its SVD,

xα = (VΣ2V T + α2VV T )−1VΣUTb

= V (Σ2 + α2I )−1ΣUTb

=n∑

i=1

φαiuTi b

σivi

where φαi =σ2i

σ2i + α2

are called filter factors

Note:

φαi =

1 if σi ασ2i

α2 σi αφTSVDi =

1 if i ≤ k0 i > k


Relation between Tikhonov and TSVD

For each k in TSVD there exists an α such that the solution to the Tikhonovproblem and the solution based on TSVD are approximately equal.


Choice of parameter α

How do we choose optimal α?

L-curve is log-log plot of the norm of the regularized solution versus the residualnorm. The best parameter lies at the corner of the L (maximum curvature)


General form of Tikhonov regularization

The Tikhonov formulation can be generalized to,

minx‖Ax − b‖22 + α2‖Lx‖2

2

where L is a discrete smoothing operator. Common choices are the discrete firstand second derivative operators.


Comparison of regularization methods

Figure 4 : The original image (top left) and blurred image (top right). Tikhonovregularization (bottom left) and TSVD (bottom right).

http://www2.compute.dtu.dk/ pcha/DIP/chap8.pdfBakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 33 / 114

Summary

Regularization suppresses components from noise and enforces regularity on thecomputed solution.

Figure 5 : Illustration of why regularization is needed


Geophysical model problem

Unknown mass with density f (t) located at depth d below the surface.

No mass outside source.

We measure vertical component gravity field, g(s),

Figure 6 : Gravity surveying example problem.



Magnitude of gravity field along s is

f (t) dt

d2 + (s − t)2

and the direction is in the direction from the point at s to the point at t.

dg =sin θ

r2f (t)dt

Using sin θ = d/r and integrating we get the forward problem:

g(s) =

∫ 1

0

d

(d2 + (s − t)2)3/2f (t)dt



Swapping elements of forward problem, we get the inverse problem.∫ 1

0

d(d2 + (s − t)2

)3/2︸︷︷︸K(s,t)

f (t)dt = g(s)

where f (t) is the quantity we wish to estimate given measurements of g(s).


TSVD

Figure 7 : Exact solution (bottom right) and TSVD solutions


Tikhonov regularization

Figure 8 : Exact solution (bottom right) and Tikhonov solutions


Large-scale inverse problemsSVD infeasible for large-scale problems O(N3).Apply iterative methods to the linear system

(ATA + α2I )x(α) = ATb

Generate a sequence of vectors (Krylov subspace)

Kk(ATA,ATb)def= SpanATb, (ATA)ATb, . . . , (ATA)k−1Ab

Lanczos bidiagonalization (LBD)

AVk = UkBk

ATUk = VkBTk + βkvk+1e

Tk I

UTk Uk = I and V T

k Vk = I

Bk =

α1

β1 α2

β2. . .. . . αk−1

αk

Singular vectors of Bk converge to the singular values of A. (typically largest onesconverge first)


Large-scale iterative solvers

CGLSThe LBD can be rewritten as

(ATA + α2I )Vk = Vk(BkBTk + α2I )

Find xk = Vkyk such that

yk = (BkBTk + α2I )−1‖b‖2e1

obtained by a Galerkin projection on the residual

LSQRFind xk = Vkyk by solving a k × k system of equations

yk = arg miny

‖[

Bk

βkeTk

]y − ‖b‖2e1‖2

2 + α2‖y‖22

Solve a small regularized least squares problem at each step

Additionally regularization parameter α can be estimated at each iteration.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 41 / 114

Semi-convergence behavior

Standard convergence criteria for iterative solvers based on residual do not workwell for inverse problems.

This is because measurements are corrupted by noise. Need different stoppingcriteria/ regularization methods.

From http://www.math.vt.edu/people/jmchung/resources/CSGF07.pdfBakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 42 / 114

CGLS convergence 1/2


CGLS convergence 2/2


Outline

1 Introduction







Bayes’ theorem

Reverend Thomas Bayes

Interpretation: Inductive argument

p(Hypothesis|Evidence) ∝ p(Evidence|Hypothesis)p(Hypothesis)

(left) http://www.gaussianwaves.com/2013/10/bayes-theorem/ (right) WikipediaBakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 46 / 114

Coin toss experiment: Bayesian analysis

Say we have a “biased coin”

X1,X2, . . . ,Xn+1 p(Xi = 1|π) = π p(Xi = 0|π) = 1− π

What is the probability of observing a certain sequence?

H, T, H, . . .

H, H, H, . . .

After n + 1 trials we have

p(X1 = x1,X2 = x2, . . . ,Xn+1 = xn+1|π) =n+1∏k=1

p(Xk = xk |π)

= π∑

xk (1− π)n+1−∑ xk

The Xi are conditionally independent.


Bayesian update: Uniform priorLet’s assume that we don’t have any information

p(π) =

1 0 < π < 10 otherwise


Bayesian analysis: Uniform prior

Bayes’ rule

p(π|x1, x2, . . . , xn+1) =p(x1, x2, . . . , xn+1|π)p(π)

p(x1, x2, . . . , xn+1)

Applying the Bayes rule

p(π|x1, x2, . . . , xn+1) ∝ π∑

xk (1− π)n+1−∑ xk × I0<π<1

Summary of distribution:

Conditional Mean :n

n + 2

(∑xkn

)+

1

n + 2Maximum :

∑xkn

Can approximate the distribution by a Gaussian (Laplace’s approximation)

p(π|x1, x2, . . . , xn+1) ∼ N (µ, σ2) µ =

∑xkn

σ2 =µ(1− µ)

n



Bayes’ rule

p(π|x1, x2, . . . , xn+1) =p(x1, x2, . . . , xn+1|π)p(π)

p(x1, x2, . . . , xn+1)


p(π|x1, x2, . . . , xn+1) ∝ π∑

xk (1− π)n+1−∑ xk × I0<π<1


Conditional Mean :n

n + 2

(∑xkn

)+

1

n + 2Maximum :

∑xkn


p(π|x1, x2, . . . , xn+1) ∼ N (µ, σ2) µ =

∑xkn

σ2 =µ(1− µ)

n



Bayes’ rule

p(π|x1, x2, . . . , xn+1) =p(x1, x2, . . . , xn+1|π)p(π)

p(x1, x2, . . . , xn+1)


p(π|x1, x2, . . . , xn+1) ∝ π∑

xk (1− π)n+1−∑ xk × I0<π<1


Conditional Mean :n

n + 2

(∑xkn

)+

1

n + 2Maximum :

∑xkn


p(π|x1, x2, . . . , xn+1) ∼ N (µ, σ2) µ =

∑xkn

σ2 =µ(1− µ)

n


Prior: Beta distribution

π follows a Beta(α, β) distribution

p(π) ∝ πα−1(1− π)β−1

Beta distribution is analytically tractable; example of conjugate prior.


Bayesian update: Beta prior α = 5, β = 2


Bayesian update: Beta prior α = 0.5, β = 0.5


Bayesian Analysis: Beta prior


p(π|x1, x2, . . . , xn+1) ∝ π∑

xk (1− π)n+1−∑ xk × πα−1(1− π)β−1

π∑

xk+α−1(1− π)n+1−∑ xk+β

Conditional mean

Eπ[p(π|x1, . . . , xn+1)] =

∫ 1

0

πp(π|x1, . . . , xn+1)dπ

=n

n + α + β

(∑xkn

)+

α + β

n + α + β

(α

α + β

)Observe that this gives the right limit as n→∞.


Inverse problems: Bayesian viewpointConsider the measurement equation

y = h(s) + v v ∼ N (0, Γnoise)

Notation:

y : observations or measurements - given.s : model parameters, we want to estimate.h(s) : parameter-to-observation map - given.v : additive i.i.d Gaussian noise

Using Bayes’ rule, the posterior pdf is

p(s|y) ∝ p(y |s)︸︷︷︸Data misfit

p(s)︸︷︷︸Prior

Data misfit - “How well the model reproduces data”

Prior - “Prior knowledge of unknown field ”

I Smoothness, sparsity, etc


Inverse problems: Bayesian viewpoint

Consider the measurement equation



















Geostatistical approach

Let s(x) be the parameter field we wish to recover

s(x) =∑p

k=1 fi (x)βk︸︷︷︸Deterministic term

+ ε(x)︸︷︷︸Random term

Possible choices for fi (x)

Low order polynomials f1 = 1, f2 = x , f3 = x2, etc.

Zonation modelI fi is nonzero only in certain regions

Several possible choices for ε(x)

We will assume Gaussian random fields.

Revisit this assumption (later in this talk).


Gaussian Random Fields

GRF are multidimensional generalizations of Gaussian processes.

DefinitionA Gaussian process is a collection of random variables, any finite number of whichhave a joint Gaussian distribution.





A Gaussian process is completely specified by its mean function and covariancefunction.

µ(x)def= E[f (x)]

κ(x , y)def= E[(f (x)− µ(x))(f (y)− µ(y))]

The GP is denoted as

f (x) ∼ N (µ(x), κ(x , y))





Examples of Gaussian random fields

Figure 9 : Samples from Gaussian random fields


Geostatistical approachModel priors as Gaussian random fields

s|β ∼ N (Xβ, Γprior) p(β) ∝ 1

Posterior distributionApplying Bayes theorem

p(s, β|y) ∝ p(y |s, β)p(s|β)p(β)

exp

(−1

2‖y − h(s)‖2

Γ−1noise

− 1

2‖s − Xβ‖2

Γ−1prior

)

Maximum a posteriori (MAP) estimate:

s, β = arg mins,β

− log p(s, β|y)

= arg mins,β

1

2‖y − h(s)‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior






exp

(−1

2‖y − h(s)‖2

Γ−1noise

− 1

2‖s − Xβ‖2

Γ−1prior

)


s, β = arg mins,β

− log p(s, β|y)

= arg mins,β

1

2‖y − h(s)‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior






exp

(−1

2‖y − h(s)‖2

Γ−1noise

− 1

2‖s − Xβ‖2

Γ−1prior

)


s, β = arg mins,β

− log p(s, β|y)

= arg mins,β

1

2‖y − h(s)‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior


MAP Estimate - Linear Inverse Problems

Maximum a posteriori (MAP) estimate: for h(s) = Hs

s, β = arg mins,β

1

2‖y − Hs‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior

Obtained by solving the system of equations(HΓpriorH

T + Γnoise HX(HX )T 0

)(ξ

β

)=

(y0

)s = X β + ΓpriorH

T ξ

Solved using a matrix-free Krylov solver.

Requires fast ways to compute Hx and Γpriorx

Preconditioner21 using a low-rank representation of Γprior

21Preconditioned iterative solver developed in Saibaba and Kitanidis, WRR 2012.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 58 / 114



s, β = arg mins,β

1

2‖y − Hs‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior



)(ξ

β

)=

(y0


T ξ







s, β = arg mins,β

1

2‖y − Hs‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior



)(ξ

β

)=

(y0


T ξ





Interpolation using Gaussian Processes22

The posterior is Gaussian with

µpost(x∗) = κ(x∗, x)(κ(x, x) + σ2I )−1y(x)

covpost(x∗, x∗) = κ(x∗, x∗)− κ(x∗, x)(κ(x, x) + σ2I )−1κ(x, x∗)

22Gaussian Processes for Machine Learning, Rasmussen and WilliamsBakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 59 / 114

Application: CO2 monitoring

Challenge:

Real-time monitoring of CO2 concentration

Time series of noisy seismic traveltime tomography data.

288 measurements and 234× 217 unknowns

A.K. Saibaba, Ambikasaran, Li, Darve, Kitanidis, Oil and Gas Science and Technology 67.5(2012): 857.


Matern covariance family

Matern class of covariance kernels

κ(x , y) =(αr)ν

2ν−1Γ(ν)Kν(αr), α > 0, ν > 0

Here, r = ‖x − y‖2 is the radial distance between points x and y .

Examples: Exponential kernel (ν = 1/2), Gaussian kernel ν =∞.


Matern covariance kernels

Deconvolution equation

y(t) =

∫ T

0

f (t − τ)s(τ)dτ

Matern covariance kernels ν = 1/2, 3/2,∞

κ(x , y) = exp(−|x − y |/L)




y(t) =

∫ T

0



κ(x , y) = (1 +√

3|x − y |/L) exp(−√

3|x − y |/L)




y(t) =

∫ T

0



κ(x , y) = exp(−|x − y |2/L2)


Fast covariance evaluations

Consider the Gaussian priors

s|β ∼ N (Xβ, Γprior)

Covariance matrices are dense - expensive to store and compute.

For example, a dense 106 × 106 matrix costs 7.45 TB.

Typically, only need to evaluate Γpriorx and Γ−1priorx .





Standard approaches

FFT based methods,

Fast Multipole Method,

Hierarchical Matrices

Kronecker tensor product approximations.





Standard approaches

FFT based methods,

Fast Multipole Method,

Hierarchical Matrices

Kronecker tensor product approximations.

Compared to the naive O(N2)

Storage cost: O(N logα N) Matvec cost: O(N logβ N)


Toeplitz Matrices

A Toeplitz matrix T is an N × N matrix with entries such that Tij = ti−j , i.e. amatrix of the form

T =

t0 t−1 t−2 . . . t−(N−1)

t1 t0 t−1

t2 t1 t0

......

. . .

tN−1 . . . t0

Suppose points xi = i × h and yj = j × h for i , j = 1, . . . ,N

Stationary kernels Qij = κ(xi , yj) = κ((i − j)h)

Translation-invariant kernels Qij = κ(xi , yj) = κ(|i − j |h)

Need to store only O(N) entries, compared to O(N2) entries.


FFT based methods

Toeplitz matrices arise from stationary covariance kernels on regular grids

c b ab c ba b c

Periodic embedding=⇒

c b a a bb c b a aa b c b aa a b c bb a a b c

Diagonalizable by Fourier basis

Matrix-Vector Products for Toeplitz matrices O(N logN)

Restricted to regular, equispaced grids.


H-matrix formulation: An Intuitive Explanation.

Consider for xi , yi = (i − 1) 1N−1 , i = 1, . . . ,N

κα(x , y) =1

|x − y |+ αα > 0

Figure 10 : blockwise rank-ε α = 10−6, ε = 10−6,N = M = 256


H-matrix formulation: An Intuitive Explanation.

Consider for xi , yi = (i − 1) 1N−1 , i = 1, . . . ,N

κ(x , y) = exp(−|x − y |)

Figure 11 : blockwise rank-ε ε = 10−6,N = M = 256


Exponentially decaying singular values of off-diagonalblocks

κα(x , y) =1

|x − y |+ αα > 0 (1)

Figure 12 : First 32 singular values of off-diagonal sub-blocks of matrix corresponding tonon-overlapping segments (left) [0, 0.5] × [0.5, 1] and (right) [0, 0.25] × [0.75, 1.0]

The decay of singular values can be related to the smoothness of the kernel.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 68 / 114

Prof. SVD - Gene Golub


Prof. SVD - Gene GolubRank-10 approximation






Hierarchical-matrices24

Hierarchical separation of space.

Low rank sub-blocks with well separated clusters.

Mild restrictions on the types of permissible kernels

Level 0

Level 1

Level 2

Level 3

Full-rank blocks Low-rank blocks

24Hackbusch - 2000, Grasedyck and Hackbusch - 2003, Bebendorf - 2008Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 70 / 114





Level 0

Level 1

Level 2

Level 3







Level 0

Level 1

Level 2

Level 3







Level 0

Level 1

Level 2

Level 3







Level 0

Level 1

Level 2

Level 3



Clustering


Block clustering





Quasi-linear geostatistical approach

Maximum a posteriori estimate:

arg mins,β

1

2‖y − h(s)‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior

Algorithm 1 Quasi-linear geostatistical approach

1: while Not converged do2: Solve the system of equations26 ,(

JkΓpriorJTk + Γnoise JkX

(JkX )T 0

)(ξk+1

βk+1

)=

(y − h(sk) + Jksk

0

)where, the Jacobian J = ∂h

∂s

∣∣s=sk

3: The update sk+1 = Xβk+1 + ΓpriorJTk ξk+1

4: end while


Quasi-linear geostatistical approach

Maximum a posteriori estimate:

arg mins,β

1

2‖y − h(s)‖2

Γ−1noise

+1

2‖s − Xβ‖2

Γ−1prior

Algorithm 2 Quasi-linear geostatistical approach

1: while Not converged do2: Solve the system of equations26 ,(

JkΓpriorJTk + Γnoise JkX

(JkX )T 0

)(ξk+1

βk+1

)=

(y − h(sk) + Jksk

0

)where, the Jacobian J = ∂h

∂s

∣∣s=sk

3: The update sk+1 = Xβk+1 + ΓpriorJTk ξk+1

4: end while


MAP Estimate - Quasi-linear Inverse Problems

At each step,

linearize to get a local Gaussian approximation

Solve a sequence of linear inverse problems.



At each step,





At each step,





At each step,




Non-Gaussian priors

Gaussian random fields often produce smooth reconstructions

Often need discontinuous reconstructionsI Facies detection, tumor location.

Several possibilities

Total Variation regularization

Level Set approach

Markov Random Fields

Wavelet based reconstructions

Only scratching the surface, lots of techniques (and literature) available.


Non-Gaussian priors

Gaussian random fields often produce smooth reconstructions

Often need discontinuous reconstructionsI Facies detection, tumor location.

Several possibilities

Total Variation regularization

Level Set approach

Markov Random Fields

Wavelet based reconstructions

Only scratching the surface, lots of techniques (and literature) available.


Total variation regularization

Total variation in 1D

TV (f ) = supn−1∑k=1

|f (xk+1)−f (xk)|

Measure of arc length of a curve

Gif: Wikipedia, Figure: Kaipio et al. Statistical and computational inverse problems. Vol.160. Springer Science & Business Media, 2006


Total Variation RegularizationMAP estimate (penalize discontinuous changes)

mins

1

2‖y − h(s)‖2

Γ−1noise

+ α

∫Ω

|∇s|ds |∇s| ≈√∇s · ∇s + ε

Figure 13 : Inverse Wave propagation problem. (left) Cross-sections of inverted andtarget models, (right) Surface model of the target.

Akcelic, Biros and Ghattas, Supercomputing, ACM/IEEE 2002 Conference. IEEE, 2002.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 78 / 114

Level Set approach

s(x) = cf (x)H(φ(x)) + cb(x)(1− H(φ(x))) H(x) =1

2(1 + sign(x))

Figure 14 : Image courtesy of Wikipedia

Topologically flexible - able to recover multiple connected components

Evolve the shape by the minimizing an objective function.


Bayesian Level set approachLevel set function

s(x) = cf (x)H(φ(x)) + cb(x)(1− H(φ(x))) H(x) =1

2(1 + sign(x))

Employ a Gaussian random field as prior for φ(x)

Groundwater flow

−∇ · κ∇u(x) = f (x) x ∈ Ω

u = 0 x ∈ ∂Ω

Transformation s = log κ

Iglesias et al. Preprint arXiv:1504.00313Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 80 / 114

Outline

1 Introduction







Tracking trajectories


Tracking trajectories


360 panorama - Teliportme


4D Var FilteringConsider the dynamical system

∂v

∂t= F (v) + η

v(x , 0) = v0(x)

3D Var Filtering

J3(v)def= ‖y(T )− h(v(x ;T ))‖2

Γ−1noise

+1

2‖v0(x)− v∗0 (x)‖2

Γ−1prior

Optimization problem

v0def= arg min

v0

Jk(v) k = 3, 4

4D Var Filtering

J4(v)def=

Nt∑k=1

‖y(tk)− h(v(x ; tk))‖2Γ−1

noise

+1

2‖v0(x)− v∗0 (x)‖2

Γ−1prior


Application: Contaminant source identification

Transport equations

∂c

∂t+ v · ∇c = D∇2c

D∇c · n = 0

c(x , 0) = c0(x)

Estimate initial conditions from measurements of the contaminant field.

Akcelik, Volkan, et al. Proceedings of the 2005 ACM/IEEE conference on Supercomputing.IEEE Computer Society, 2005.


Linear Dynamical System

System Noise:

Measurements:

uk−1 uk uk+1

· · · sk−1 F sk F sk+1 · · ·

vk−1 Hk−1 vk Hk vk+1 Hk+1

yk−1 yk yk+1


State Evolution equations

Linear evolution equations

sk+1 = Fksk + uk uk ∼ N (0, Γprior)

yk+1 = Hk+1sk+1 + vk vk ∼ N (0, Γnoise)

obtained by discretizing a PDE

Nonlinear evolution equations

sk+1 = f (sk) + uk uk ∼ N (0, Γprior)

yk+1 = h(sk+1) + vk vk ∼ N (0, Γnoise)

Can be linearized (Extended Kalman Filter) or handled as is (Ensemble filtering)


State Evolution equations

Linear evolution equations



obtained by discretizing a PDE

Nonlinear evolution equations

sk+1 = f (sk) + uk uk ∼ N (0, Γprior)

yk+1 = h(sk+1) + vk vk ∼ N (0, Γnoise)

Can be linearized (Extended Kalman Filter) or handled as is (Ensemble filtering)


Kalman Filter

Current N (sk|k,Σk|k) Update PredictFuture

N (sk+1|k+1,Σk+1|k+1)

Transition matrix Fk Observation Hk

Sys. noisewk ∼ N (0,Γsys)

Meas. noisevk ∼ N (0,Γnoise)

All variables are modeled as Gaussian random variables

Completely specified by the mean and covariance matrix.

Kalman filter provides a recursive way to update state knowledge andpredictions.


Standard implementation of Kalman Filter

Predict

sk+1|k = sk|k −Σk+1|k = FkΣk|kF

Tk + Γprior O(N3)

Update

Sk = HkΣk+1|kHTk + Γnoise O(nmN

2)

Kk = Σk+1|kHTS−1

k O(nmN2)

sk+1|k+1 = sk+1|k + Kk(yk − Hk sk+1|k) O(nmN)

Σk+1|k+1 = (Σ−1k+1|k + HT

k Γ−1noiseHk)−1 O(nmN

2 + N3)

N: number of unknowns and nm: number of measurements

Storage cost O(N2) and computational cost O(N3)

This cost is prohibitively expensive for large-scale implementation


Standard implementation of Kalman Filter

Predict

sk+1|k = sk|k −Σk+1|k = FkΣk|kF

Tk + Γprior O(N3)

Update

Sk = HkΣk+1|kHTk + Γnoise O(nmN

2)

Kk = Σk+1|kHTS−1

k O(nmN2)

sk+1|k+1 = sk+1|k + Kk(yk − Hk sk+1|k) O(nmN)

Σk+1|k+1 = (Σ−1k+1|k + HT

k Γ−1noiseHk)−1 O(nmN

2 + N3)

N: number of unknowns and nm: number of measurements

Storage cost O(N2) and computational cost O(N3)

This cost is prohibitively expensive for large-scale implementation


Ensemble Kalman Filter

The EnKF is a Monte Carlo approximation of the Kalman filter.

Ensemble of state variables: X = [x1, . . . , xN ]

Ensemble of realizations are propagated individuallyI Can reuse legacy codesI Easily parallelizable

To update filter compute statistics based on the ensemble

Unlike Kalman filter, can be readily applied to nonlinear problems

The ensemble mean and covariance can be computed as

E[X ] =1

N

N∑k=1

xk C =1

N − 1AAT

where A is the mean subtracted ensemble.


Ensemble Kalman Filter

The EnKF is a Monte Carlo approximation of the Kalman filter.

Ensemble of state variables: X = [x1, . . . , xN ]

Ensemble of realizations are propagated individuallyI Can reuse legacy codesI Easily parallelizable

To update filter compute statistics based on the ensemble

Unlike Kalman filter, can be readily applied to nonlinear problems

The ensemble mean and covariance can be computed as

E[X ] =1

N

N∑k=1

xk C =1

N − 1AAT

where A is the mean subtracted ensemble.


Application: Real-time CO2 monitoring

Sources

Receivers

Sources fire a pulse, receivers measure time delay.

Measurements - travel time of each source-receiver pair.I 6 sources, 48 receivers = 288 measurements

Assumption: rays travel in straight-line path

tsr =

∫ recv

source

1

v(x)︸︷︷︸Slowness

d`+ noise

Model problem for: reflection seismology, CT scanning, etc.


Random Walk Forecast model

Evolution of CO2 can be modeled as



Random walk assumption Fk = I

Useful modeling assumption when measurements can be acquired rapidly

Applications: Electrical Impedance Tomography, Electrical ResistivityTomography, Seismic Travel-time tomography

Treat Γprior using Hierarchical matrix approach

A.K. Saibaba, E.L. Miller, P.K. Kitanidis, A Fast Kalman Filter for time-lapse ElectricalResistivity Tomography. Proceedings of IGARSS 2014, Montreal


Random Walk Forecast model

Evolution of CO2 can be modeled as



Random walk assumption Fk = I

Useful modeling assumption when measurements can be acquired rapidly

Applications: Electrical Impedance Tomography, Electrical ResistivityTomography, Seismic Travel-time tomography

Treat Γprior using Hierarchical matrix approach

A.K. Saibaba, E.L. Miller, P.K. Kitanidis, A Fast Kalman Filter for time-lapse ElectricalResistivity Tomography. Proceedings of IGARSS 2014, Montreal


Results: Kalman Filter

Figure 15 : True and estimated CO2-induced changes in slowness (reciprocal of velocity)between two wells for the grid size 234 × 219 at times 3, 30 and 60 hours respectively.


Comparison of costs of different algorithms

Grid size 59× 55

Γprior is constructed using kernel κ(r) = θ exp(−√r)

Γnoise = σ2I with σ2 = 10−4

Saibaba, Arvind K., et al. Inverse Problems 31.1 (2015): 015009.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 94 / 114

Error in the reconstruction

Γprior is constructed using kernel κ(r) = θ exp(−√r)

Γnoise = σ2I with σ2 = 10−4


Conditional Realizations

Figure 16 : Conditional realizations of CO2-induced changes in slowness (reciprocal ofvelocity) between two wells for the grid size 59 × 55 at times 3, 30 and 60 hoursrespectively.


Outline

1 Introduction










Notation:

y : observations or measurements - given.s : model parameters, we want to estimate.h(s) : parameter-to-observation map - given.v : additive i.i.d Gaussian noise




The posterior distribution is the Bayesian solution to the inverse problem.


Bayesian Inference: Quantifying uncertaintyMaximum-a-posteriori (MAP) estimate arg max p(s|y)

Conditional mean

sCM = Es|y [s] =

∫s p(s|y)ds

Credibility intervals: Find sets C (y)

p[s ∈ C (y)|y ] = 1− α

Sample realizations from the posterior


Bayesian Inference: Quantifying uncertaintyMaximum-a-posteriori (MAP) estimate arg max p(s|y)

Conditional mean

sCM = Es|y [s] =

∫s p(s|y)ds

Credibility intervals: Find sets C (y)

p[s ∈ C (y)|y ] = 1− α

Sample realizations from the posterior


Linear Inverse ProblemsRecall the distribution is given by

p(s|y) ∝ exp

(−1

2‖y − Hs‖2

Γ−1noise

− 1

2‖s − µ‖2

Γ−1prior

)

Posterior distribution

s|y ∼ N (sMAP, Γpost)

Γpost =(

Γ−1prior + HTΓ−1

noiseH)−1

= Γprior − ΓpriorHT (HΓpriorH

T + Γnoise)−1HΓprior

sMAP = Γpost(HTΓ−1

noisey + Γ−1priorµ)

Observe thatΓpost ≤ Γprior


Application: CO2 monitoring

Variance = diag(Γpost) = diag(Γ−1prior + HTΓ−1

noiseH)−1

A.K. Saibaba, Ambikasaran, Li, Darve, Kitanidis, Oil and Gas Science and Technology 67.5(2012): 857.


Nonlinear Inverse ProblemsLinearize the forward operator (at the MAP point)

h(s) = h(sMAP) +∂h

∂s(s − sMAP) +O(‖s − sMAP‖2

2)

Groundwater flow equations

−∇ · (κ(x)∇φ) = Qδ(x− xsource) x ∈ Ω

φ = 0 x ∈ ΩD

Inverse problem:

Estimate hydraulic tomography κ from discrete measurements of φ.

To make problem well-posed, work with s = log κ.

Saibaba, Arvind K., et al., Advances in Water Resources 82 (2015): 124-138.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 102 / 114

Nonlinear Inverse Problems

Linearize the forward operator (at the MAP point)

h(s) = h(sMAP) +∂h

∂s(s − sMAP) +O(‖s − sMAP‖2

2)

Figure 17 : (left) Reconstruction of log conductivity (right) Posterior variance

Saibaba, Arvind K., et al., Advances in Water Resources 82 (2015): 124-138.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 102 / 114

Monte Carlo samplingSuppose X has density p(x) and we are interested in f (X )

E[f (X )] =

∫f (x)p(x)dx = lim

N→∞1

N

N∑k=1

f (xk)

Approximate using sample averages

E[f (X )] ≈ 1

N

N∑k=1

f (xk)

p(x) is understood to be the posterior distribution.

If samples are easy to generate, procedure is straightforward.

Use Central Limit Theorem to generate confidence intervals.

Generating samples from p(x) may not be straightforward.


Acceptance-rejection samplingApproximate distribution by an easier distribution

Points under curve

Points generated× box area = lim

n→∞

∫ B

A

f (x)dx

From PyMC2 website: http://pymc-devs.github.io/pymc/theory.htmlBakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 104 / 114

Markov chains

Consider a sequence of random variables X1,X2, . . .

p(Xt+1 = xt+1|Xt = xt , . . . ,X1 = x1) = p(Xt+1 = xt+1|Xt = xt)

The future depends only on the present - not the past!

Under some conditions, the chain has a stationary distribution.


Implementation

Create a Markov Chain whose stationary distribution is p(x)

1 Draw a proposal y from q(y |xn)

2 Calculate acceptance ratio

α(xn, y) = min

1,

p(y)q(xn|y)

p(xn)q(y |xn)

3 Accept/Reject

xn+1 =

y with probabilityα(xn, y)xn with probability 1− α(xn, y)

If q(x , y) = q(y , x) then α(xn, y) = min1, p(y)/p(xn)


MCMC demo

Demo at: http://chifeng.scripts.mit.edu/stuff/mcmc-demo/


http://chifeng.scripts.mit.edu/stuff/mcmc-demo/

Properties of MCMC sampling

Ergodic theorem for expectations

limN→∞

1

N

N∑k=1

f (xi ) =

∫Ω

f (x)p(x)dx

However samples xk are no longer i.i.d. Has higher variance than MC sampling.

Popular sampling strategies

Metropolis-Hastings

Gibbs samplers

Hamiltonian MCMC

Adaptive MCMC with Delayed rejection (DRAM)

Metropolis adjusted Langevin Algorithm (MALA)


Curse of dimensionality

What is the probability of hitting a hypersphere inscribed in a hypercube?

In dimension n = 100, the probability < 2× 10−70.


Stochastic Newton MCMC

Martin, James, et al. SISC 34.3 (2012): A1460-A1487.Bakhos, Kitanidis, Saibaba Large-Scale Inverse Problems June 28, 2015 110 / 114

Outline

1 Introduction







Opportunities

Theoretical and numerical

“Big data” meets “Big Models”

Model reduction

Posterior uncertainty quantification

Applications

New application areas, new technologies that generate inverse problems

Combining multiple modalities to make better predictions

Software that transcends application areas


Resources for learning Inverse ProblemsBooks

Hansen, Per Christian. Discrete inverse problems: insight and algorithms.Vol. 7. SIAM, 2010.

Hansen, Per Christian. Rank-deficient and discrete ill-posed problems:numerical aspects of linear inversion. Vol. 4. SIAM, 1998.

Hansen, Per Christian, James G. Nagy, and Dianne P. O’Leary. Deblurringimages: matrices, spectra, and filtering. Vol. 3. Siam, 2006.

Tarantola, Albert. Inverse problem theory and methods for model parameterestimation. SIAM, 2005.

Kaipio, Jari, and Erkki Somersalo. Statistical and computational inverseproblems. Vol. 160. Springer Science & Business Media, 2006.

Vogel, Curtis R. Computational methods for inverse problems. Vol. 23.SIAM, 2002.

PK Kitanidis. Introduction to geostatistics: applications in hydrogeology.Cambridge University Press, 1997.

Cressie, Noel. Statistics for spatial data. John Wiley & Sons, 2015.


Resources for learning Inverse Problems

Software Packages

Regularization Tools (MATLAB)I Website:

http://www2.compute.dtu.dk/~pcha/Regutools/regutools.html

PESTI Website http://www.pesthomepage.org/

bgaPESTI Website: http://pubs.usgs.gov/tm/07/c09/

MUQI Website: https://bitbucket.org/mituq/muq


http://www2.compute.dtu.dk/~pcha/Regutools/regutools.html

http://www.pesthomepage.org/

http://pubs.usgs.gov/tm/07/c09/

https://bitbucket.org/mituq/muq

Documents

Large-Scale Inverse Problems