Ill-Posedness and Regularization of Linear Operators (1 lecture)

1IPIM, IST, José Bioucas, 2007 IPIM, IST, José Bioucas, 2007

Ill-Posedness and Regularization of Linear Operators (1 lecture)

Singular value decomposition (SVD) in finite-dimensional spaces

Least squares solution; Moore-Penrose pseudo inverse

Geometry of a linear inverse

Ill-posed and ill-conditioned problems

Tikhonov regularization; Truncated SVD

SVD of compact operators

2IPIM, IST, José Bioucas, 2007

Basics of linear operators in function spaces (Appendix B of [RB1])

Operator A from to is a mapping that assigns to each (domain) an element (range)

Hilbert spaces (finite or infinite)

A is defined every where

A is on operator on

Linear operators

We write

Null space of a linear operator (it is a subspace)


Basics of linear operators in function spaces (Apendix B of [RB1])

A linear operator A is continuous iff it is bounded

Adjoint operator depends on the inner product. Examples:

A is bounded if there exists a constant M:

is a matrix and

Norm of A:

Adjoint operator: is the unique operator such that

is a matrix and

Note:


Eigenvalues and eigenvectors of symmetric matrices

symmetric (self-adjoint )

is equipped with the standard Euclidian inner product

Eigen-equation

are real

may always be chosen to form an orthogonal basis

Let

(U is an unitary matrix)

Eigen-equation in terms of U


Spectral representation of symmetric matrix A

1. projects the input vector along

2. synthetizes by the linear combination

Action of a real symmetric matrix on an input vector


Functions of a symmetric matrix

From the following properties of an unitary matrix:

1.

2.

3.

It follows that

3. If A is non-singular

2. If h(A) is a power series

1.

4. If (A is positive semi-positive - PSD), we can define


• but A is not self-adjoint ( ), the eigendecomposition does not have the nice properties of self-adjoint matrices. The cyclic matrices are an exception

Singular value decomposition of a real (complex) rectangular matrix

• the eigenvalue problem is meaningless. The Singular value decomposition provides a generalization of the self-adjoint spectral decomposion

equipped with the standard Euclidian inner product


Singular value decomposition

and are equipped with the standard Euclidian inner products

isometric

left singular vectors

right singular vectors

singular values


matrix norms:

Singular value decomposition: consequences

range and null-space of

range and null-space of


Singular value decomposition

Action of a real symmetric matrix on the vector

1. projects the input vector along

2. synthetizes by the linear combination


Singular value decomposition: illustration

50 100 150 200 250

20

40

60

80

100

1200

0.2

0.4

0.6

0.8

1

0 50 100 150-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

u1

u10

0 50 100 15010

-20

10-15

10-10

10-5

100

105

i

0 50 100 150 200 250 300-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

v1

v10


Inversion methods:

A is not invertible

b)a)

A is invertible:


Least-squares approach

Orthogonal components


Generalized inverse

is the least-squares solution of minimum-norm, orthe generalized solution


Moore-Penrose pseudo-inverse and Minimum-norm solution

Moore-Penrose pseudo-inverse (r = n · m

Minimum-norm solution (r = m < n


Moore-Penrose pseudo-inverse: a variational point of view

is invertible

Minimization of the observed data misfit

Normal equations


Effect of noise

The boundary of is an ellipse centered at with principal axesaligned with . The lenght of the k-th principal semi-axis is


corresponding singular vectors

Classification of the linear operators

• If n m A is Ill-posed

• In any case “small” singular values are sources of instabilities.

Often, the smaller the eigenvalues the more oscilating the (high frequences)

Regularization: shrink/threshold large values of

i.e, multiply the eigenvalues by a regularizer function such that

as


Regularization

1)2 ) The larger singular are retained

as

Regularization by shrinking/thresholding the spectrum of A

Such that

Truncated SVD (TSVD)

Tikhonov (Wiener) regularization


Regularization by shrinking/thresholding the spectrum of A

Tikhonov regularization: variational formulation

TSVD

Tikhonov

unitary

Lets write the singular decomposition of A as


Tikhonov regularization

Family of quadratic regularizers

Does SVD plays a role?

Thus, the Tikhonov regularized solution is given by

which is the solution of the variational problem

for any Hilbert spaces (see Appendic E of [RB 1])



50 100 150 200 250

20

40

60

80

100

1200

0.2

0.4

0.6

0.8

1

0 50 100 15010

-20

10-15

10-10

10-5

100

105

i

0 50 100 150 200 250 3000

0.2

0.4

0.6

0.8

1

1.2

1.4

f

g



0 50 100 150 200 250 300-100

-50

0

50

100

150

f

f

0 50 100 150 200 250 300-0.2

0

0.2

0.4

0.6

0.8

1

1.2

f

f

0 50 100 150 200 250 300-0.2

0

0.2

0.4

0.6

0.8

1

1.2

f

f

0 50 100 150 200 250 300-0.2

0

0.2

0.4

0.6

0.8

1

1.2

f

f



0 20 40 60 80 100 120 14010

-20

10-15

10-10

10-5

100

i

uk*w

0 50 100 150 200 250 300-20

-15

-10

-5

0

5

10

f

f

0 50 100 150 200 250 300-1.5

-1

-0.5

0

0.5

1

1.5

2

f

f

0 50 100 150 200 250 300-0.2

0

0.2

0.4

0.6

0.8

1

1.2

f

f



0 5 10 15 20

100.3

100.5

100.7

||fe-f||

10-6

10-4

10-2

100

102

100

101

102

L-curve


Medium/large systems

For medium/large systems, the SVD is impracticable

Example: Landweber iterations

The optimization problem

is solved by resorting to iterative methods that depend only onThe operators

with the Euler-Lagrange equation


3. The singular values are positive real numbers and are in nonincreasing order,

Singular system for a compact linear operator ( areHilbert spaces) is a countable set of triples with the following Properties:

Singular value decomposition in infinite-dimensional spaces

1. The right singular vectors forms an orthonormal basis for

2. The left singular vectors form an orthonormal basis for the closure of

4. For each j,

5. If is infinite dimensional,

6. A has the representation


Example of compact operators

1. Any linear operator for which is finite dimensional is compact

2. The diagonal operator on

3. The Fredholm first kind integral on (the space of real-valued square integrable functions on - a Hilbert space )


Compact linear operators in infinite dimensional spaces are ill-posed

be a compact linear operator are infinite dimensionalHilbert spaces.

1. If is infinite dimensional, then the operator equation is ill-posed in the sense that

•

• The solution is not stable

2. If is finite dimensional then the solution is not unique


Summary: SVD/least-squares based solutions

Least-squares approach

Minumum-norm solution


Truncated SVD (TSVD)

Tikhonov (Wiener) regularization

Summary: Regularized solutions

which is the solution of the variational problem

Regularizer(Penalizing function)


Summary: Medium/large systems with quadratic regularization

For medium/large systems, the SVD is impracticable. (periodic convolution operators are an important exception)

Example: Landweber iterations

The optimization problem

is solved by resorting to iterative methods that depend only on

with the Euler-Lagrange equation

Quadratic regularizer


Summary: Non-quadratic regularization

Example: discontinuity preserving regularizer

penalizes oscillatory solutions


0 50 100 150 200 250 30010

-20

10-15

10-10

10-5

100

105

vk*g

0 50 100 150 200 250 300-0.2

0

0.2

0.4

0.6

0.8

1

1.2

f

g

50 100 150 200 250

50

100

150

200

250

Matrix A

Example: deconvolution of a step


0 50 100 150 200 250 300-0.2

0

0.2

0.4

0.6

0.8

1

1.2

f

fTSVD

fTikhonov

fTikhonov(D)

fTV

Example: deconvolution of a step


Example: Sparse reconstruction

100 200 300 400 500 600 700 800 900 1000

20

40

60

80

100120

0 200 400 600 800 1000 1200-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Original data - f

0 20 40 60 80 100 120 140-0.1

-0.05

0

0.05

0.1

0.15

Observed data - g

norm


Example: Sparse reconstruction

0 200 400 600 800 1000 1200-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

f

fpseudo-inv

0 200 400 600 800 1000 1200-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

f

f

Pseudo-inverse regularization


Example: Sparse reconstruction. MM optimizationnorm

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

0.5

1

1.5

2

2.5


[Ch9.; RB1], [Ch2,Ch3; L1]

Majorization Minimization [PO1], [PO3] Compressed Sensing [PCS1]

Bibliography

Important topics

Matlab scripts

TSVD_regularization_1D.m TSVD_Error_1D.m step_deconvolution.m l2_l1sparse_regression.m

Documents

Ill-Posedness and Regularization of Linear Operators (1 lecture)