Computers Elect. Engng Vol. 18, No. 6, pp. 499-505, 1992 0045-7906/92 $5.00 + 0.00 Printed in Great Britain Pergamon Press Ltd

S P A R S E M A T R I X T E C H N I Q U E S A P P L I E D T O

D E C O N V O L U T I O N

GREG COCHRAN

Logicon R&D Associates, 5025 Windward Circle, Colorado Springs, CO 80917, U.S.A.

(Received for publication 6 October 1991; accepted in final revised form 12 February 1992)

Abstract--Sparse matrix techniques are applied to the problem of image deconvolution. This method allows the solution of practical deconvolution problems with spatially varying PSFs in a computationally efficient manner. This paper contains a description of this algorithm, its performance in terms of computational requirements, and presents results for sample calculations.

1. I N T R O D U C T I O N

The signal received on a focal plane can be modelled by the blurring of an original signal, o(x), by a general function h(x, y). The functional dependence implies that the received signal, g(x), is spread our over the focal plane according to the convolution of h(x, y) and o(x), with noise or distortion added. We assume linear distortion, so the blurred information, denoted by g(x), can be expressed by the superposition integral:

g(x) = f f o(y)h(x ,y) dy + c(x). (1)

Here c(x) is a noise or distortion term. We would like to calculate a linear estimate of the true image from the measurements, knowledge

of the PSF, and the statistics of the noise. We could approach this problem by restating the integral as a matrix equation:

g = Ho + c. (2)

Here g is the n-vector of measurements, c is the n-vector of noise, o is the m vector of the original signal, and H is the n x m matrix that performs the linear superposition (the elements of H can be determined from the PSF evaluated at the grid points). If we define the noise covariance matrix by

R = (cTc) (3)

(here ( ) denotes expectation operator) then the noise-weighted least square estimate, 6, is given by the equation:

6 = (H TR-I H) -j H TR-I O. (4)

For convenience, we will define the A matrix by the following equation:

A = (H T R - ' H). (5)

This formulation allows linear estimation of the true signal with a completely general PSF. The matrix inversion (or factorization) required for this method, the inversion of A, is in general very computationally demanding.

For a general matrix, this inversion will require 1/2n3add/multiply pairs, where n is the dimension of the matrix. Since n is the number of pixels, and may range from thousands to millions, this general algorithm is impractical for all but small cases (Ref. [1]).

Typically a special case is solved. By far the most common special case is that for which the PSF is shift invariant. This implies that the PSF reduces to the special form:

h (x, y) = h (x - y). (6)

499

500 GREG COCHRAN

When equation (6) is substituted into equation (1), the right hand side assumes the form of a convolution integral. Since convolution reduces to multiplication in Fourier space, equation (1) takes the form:

G(u) = F(u )H(u ) + C(u) (7)

where G, F, H and C refer to the Fourier transforms of the functions g,f , h and c. Since the Fourier transforms can be evaluated in n log n operations, linear image estimation for translationally invariant PSFs can be performed in n log n operations, a tremendous improvement over the general case. Unfortunately, this method fails in cases where the PSF varies.

For many cases, however, another simplification is possible, even if the PSF varies. As long as the PSF does not reach out too far (small compared with the grid size) we can use a sparse matrix approach.

For this case, the H matrix [as defined in equation (2)] will be sparse. That is, most of the entries in H will be zero. If the PSF is sufficiently narrow, this will also be true for A, when R is diagonal. (A diagonal R corresponds to a delta-correlated noise, such as shot noise.) The (i , j)th element of the A matrix is nonzero only if there is a point reached by the point spread function of i and the point spread function of j . By taking advantage of this sparseness, it is possible to greatly reduce the required calculations. The general approach is to solve equation (4) by performing a Cholesky factorization of the A matrix, while using our freedom to reorder the variables (since the matrix is positive definite, no pivoting is necessary) to minimize the required number of operations. For the case we are examining, the number of operations using this approach scales as n to the 1.5 power, rather than n 3 as in the general case (Refs [2, 3]). This means that we can solve significant image recovery problems even if the PSF varies. In the following sections, we will explain the details of this method. Section 2 explains Cholesky decomposition. Section 3 explains the importance of a good ordering of the variables, and Section 4 describes an efficient ordering method called nested dissection. In Section 5, we show some results of applying this method to simulated data, and in Section 6 we mention some other possible applications.

2. C H O L E S K Y D E C O M P O S I T I O N

Cholesky decomposition is a method for solving linear equations without explicit matrix inversion. If the problem is properly prepared, this method allows us to take advantage of the sparsity of the problem in order to economize on mathematical operations. This section explains what a Cholesky decomposition of a matrix is and how it is calculated.

Let us first define certain quantities in order to recast the problem in a standard notation. We define the vector b by the equation:

b = H T R - Io (8)

the matrix A by the equation:

A = H T R - i H (9)

and the vector x by the equation:

x = 6. ( 1 0 )

Then equation (4) can be rewritten as:

Ax ~- b. (11)

We recall that A is a square sparse matrix. A is an n x n matrix, where n is the number of pixels, b is an n-vector, and x, the solution vector, is also an n-vector. A is symmetric and positive definite. Because A is symmetric, applying Cholesky's method yields the triangular factorization:

A = LL T (12)

Sparse matrix techniques 501

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X X X X X X X X X X

X

X

X

X

X

X

X

X

Fig. I. Poorly ordered sparse matrix.

x

x

where L is lower triangular with positive diagonal elements. This factorization can be shown to be unique (Ref. [4]). A matrix M is lower triangular if its (i , j)th element, m0, is equal to zero if i is less than j. Using equation (12) in equation (11), we have

L L T x = b (13)

and by defining the vector w according to the equation:

w = L T x

it is clear we can obtain x by solving the triangular system:

(14)

and then the triangular system:

L T X = W. (16)

L is calculated by a procedure that is essentially equivalent to Gauss elimination. As an example of how it is done we shall explain how to calculate L T, which, of course, also gives us L. The algorithm is sketched in the following: there are n steps in the elimination. At step k, we have a matrix with zeros in rows 1 to k - 1 to the left of the diagonal and presumably nonzero elements to the left of the diagonal for rows k to n. (This condition is obviously satisfied for k = 1, i.e. for the first step. Thereafter, it will follow by induction.) In step k we seek to obtain from this matrix a modified matrix for which all the elements to the left of the diagonal in row k are also equal to zero. To accomplish this we eliminate Ak.,. by subtracting a multiple of the ith from the k th row (using Ak, i /Ai , i as the multiplying factor), beginning with i equal to one and continuing up to i equal to (k - 1). (This carries us to where we are ready to start the (k + 1)st step in our induction.) This procedure is continued with an incremental value of k, until for the last repetition k equals n. At the end of this process we have L "r, from which we automatically get L.

Solving the linear system is a simple matter once the lower triangular factor, L, of A is available. The solution of equation (14) is called forward substitution. Since L is lower triangular, the first equation has the form:

lu wl = bl (17)

L w = b (15)

502 GREG COCHRAN

X

X

X

X

X

X

X

X

X

X

X X X X X X X X X X

Fig. 2. Well ordered sparse matrix.

X

X

X

X

X

X

X

X

X

X

X X

X X

where li, j is the (i ,j)th element of the matrix L. (We know that In is not equal to zero because we have assumed that A is nonsingular.) Thus:

wl = b ~ / l l j . (18)

In solving for the next variable, we substitute the computed value of Wl, again making this an equation in one unknown that is simply solved. We can write:

121wl + 122w2 = b2 (19)

so that

w2 = [bz - 12~ ( b , / l , , )]/122. (20)

At step k we know the values of all the wis , for i < k, and have an equation in one variable, leading to the evaluation of w~. This is k continued until all the n elements of w have been computed. With w available, equation (16) can be solved in a very similar fashion. The last equation is solved first in this backward substitution. It has the form:

l , ~ x , = w , (21)

SO

x , = w,/lV~, (22)

where 1,T, is the (n, n)th element of the L v matrix. Proceeding recurrently backwards, the final solution, x~, is obtained in a very similar manner to forward substitution.

3. T H E O R D E R I N G P R O B L E M

When a lower triangular factor L is prepared from a sparse symmetric matrix A, new nonzero elements result at positions of L that correspond to zeros in A. (Cancellations may also occur, but they are rare in practice, except in some special cases. We will neglect cancellations.) In general, every element which is nonzero in A is also nonzero in L. Therefore, the nonzero pattern of L consists of all positions which contain nonzero entries in the lower triangle of A, plus the fill-in, which is the set of positions of the additional nonzero entries generated during the elimination. There are three important reasons why fill-in is undesirable: it costs memory, it increases the time required for factorization and for solution, and increases the round-off error. An important fact

Sparse matrix techniques

21 22 23 24 25 Table 1. Computational requirements, sparse deconvolution 16 17 18 19 20

Grid Ops Ops Conventional 11 12 13 14 15 size for solve for factor factor

16 32,560 614,274 2,796,202 6 7 8 9 10 20 I 11,712 3 ,157 ,062 31,850,496 24 365,932 14,978,646 362,797,056 1 2 3 4 5 45 651 ,114 31,262,378 1,383,960,937

Fig. 3. Simple ordering of pixels.

503

is that the amount of fill-in depends strongly on the order in which the variables (pixels in our case) are numbered. In the deconvolution problem, the A matrix is positive definite and pivoting is not necessary. Therefore, the order may be freely varied.

To see the potential importance of ordering, consider a matrix with full first row, first column, and main diagonal, as illustrated in Fig. 1. In this ordering of the variables, factorization following the procedure just outlined can be rather easily seen to induce nonzero entries in all possible positions of the triangular matrix. If, however, the first and last columns are interchanged and the first and last rows are also interchanged, as shown in Fig. 2, (corresponding to interchanging what we consider to be the first and last variables to be solved for) the resulting triangular factor L has no fill-in at all! This demonstrates the potential importance of a proper ordering of the pixels.

4. N E S T E D D I S S E C T I O N O R D E R I N G

Nested dissection is a method due to George [2,3] for systematically numbering a grid in a manner that partitions it and thereby reduces the computational expense associated with a Cholesky decomposition of a sparse matrix set of simultaneous equations. Measured in terms of computational load, it is the best known ordering method for the deconvolution problem. It requires the fewest operations of all ordering methods examined (for large numbers of pixels), and it is well suited to implementation via parallel processors.

The key idea of nested dissection is to find a set of pixels that cuts the focal plane roughly in half. This set of pixels is numbered last: that is, if there were 25 pixels, and a group of five pixels divides the other twenty into two "unconnected" sets, the dividing pixels will be numbered 21 through 25. We use the term "unconnected" here to mean that there are no nonzero matrix elements ag in A such that i is one set a n d j is in the other set. Here, A equals HTH or some similarly sparse matrix, as described in Section 2. This selection of a set of pixels is performed again on each of the two disjoint sets and these pixels are assigned the highest remaining numbers. This partitioning and numbering is continued until the remaining sets cannot be further partitioned. For best results, each time the set of pixels that splits (called a separator) should be chosen to have as few members as possible. Also, the disjoint set should have approximately equal numbers of elements.

9 10 25 13 14

11 12 24 15 16

18 17 23 20 19

3 4 22 7 8

1 2 21 5 6

II •

Fig. 4. Nested dissection ordering of pixels. Fig. 5. Pristine point source.

504 GF.E6 COCHRAN

I

Fig. 6. Blurred point sources (varying PSFs). Fig. 7. Restored point sources.

A 5 by 5 array ordered in a "natural" way is shown in Fig. 3, and the same array ordered by nested dissection is shown in Fig. 4.

It has been shown (Ref. [5]) for a system of sparse linear equations corresponding to a two- dimensional system with local coupling, the factorization resulting from a nested dissection ordering has an operation c o u n t O(k3n 3/2) and the solution has an operation count of size O(k2n log n), where k is the maximum number of pixels coupled by the PSF to a given pixel.

5. N U M E R I C A L R E S U L T S

This sparse matrix reconstruction concept has been tested through a simulation code. This simulation code prepares sample images and blurs them with a varying PSF. It then prepares the least squares estimator using sparse techniques and applies it to the simulated blurred data. The image estimate reconstructed using this approach is identical to the original image in the noise-free case, within roundoff error. The method can incorporate noise weighting for practical cases such as shot noise or uncorrelated detector noise. The following table shows the number of add/multiply pairs required to perform the factorization and solution using nested dissection orderings for a series of images of different sizes. For these cases, the PSF was 5 pixels wide, and was spatially variant. The number of add/multiply pairs required to factorize these problems using efficient conventional methods (1/6 n 3, for factorization of a symmetric matrix) is shown for comparison.

In Figs 5-7, we show sample results of the simulation program. The program produces an original unblurred image, a blurred image, and a reconstructed image. The PSF in this case was strongly varying, being close to a delta function (one pixel wide) in the center and 5 pixels wide at the edge. Our sample image consists of two point sources, one at the center and another at the edge with relative strengths of 1.0 and 2.5, respectively. The calculations were performed on a 24 × 24 grid. Figure 5 shows the original image. Figure 6 shows the blurred image. Figure 7 shows the reconstructed image.

6. O T H E R A P P L I C A T I O N S

This general approach should be useful in other problems other than imaging where the equivalent of the PSF varies. It may be useful for three-dimensional problems such as tomogra- phy. In a three-dimensional case, the operations required will scale as n 2 for the factorization and n 4/3 for the solution (Ref. [6]).

One-dimensional problems are especially promising. Here, both factorization and solution will scale as n (Ref. [6]), a tremendous improvement over the n cubed of the direct approach. Many problems with a varying blur should become easy.

Sparse matrix techniques 505

R E F E R E N C E S

1. R. H. T. Bates and M. J. McDonnell, Image Restoration and Reconstruction. Clarendon Press, Oxford (1986). 2. A. George, Nested dissection of a regular finite element mesh. S l A M J. numer. Anal 10, 345-363 (1973). 3. A. George and J. W. H. Liu, Computer Solution of Large Sparse Positive Definite Systems. Prentice-Hall, Englewood

Cliffs, N.J. (1981). 4. G. Strang, Linear Algebra And Its Applications. Harcourt Brace Jovanovich, San Diego (1988). 5. R. J. Lipton and R. E. Tarjan, A separator theorem for planar graphs. SIAM J. appl. Math. 36, 177-189 (1979). 6. R. J. Lipton, D. J. Rose and R. E. Tarjan, Generalized nested dissection. SIAM J. numer. Anal. 16, 346-538 (1979).