Multispectral principal component imaging

Multispectral principal component imaging Himadri S. Pal and Mark A. Neifeld

ECE and Optical Sciences Center, Univ. of Arizona, 1230 E Speedway, Tucson, AZ 85721-0104 [email protected], [email protected].

Abstract: We analyze a novel multispectral imager that directly measures the principal component features of an object. Optical feature extraction is studied for color face images, multi-spectral LANDSAT-7 images, and their grayscale equivalents. Blockwise feature extraction is performed that exploits both spatial and spectral correlation, with the goal of enhancing feature fidelity (i.e., root mean square error). The effect of varying block size, number of features, and detector noise is studied in order to quantify feature fidelity and optimize reconstruction performance. These results are compared with conventional imaging and demonstrate the advantages of the multiplexed approach. Specifically, we find that in addition to reducing the number of detectors within the imager, the reconstruction fidelity (i.e., root mean square error) can be significantly improved using a feature-specific imager.

©2003 Optical Society of America

OCIS codes: (110.0110) Imaging systems; (100.5010) Pattern recognition and feature extraction; (200.4740) Optical processing; (100.3010) Image reconstruction techniques

References and Links 1. C. J. Oliver, “Optical image processing by multiplex coding,” Appl. Opt. 15, 93–106 (1976). 2. M. P. Christensen, G. W. Euliss, M. J. McFadden, K. M. Coyle, P. Milojkovic, M. W. Haney, J. van der

Gracht, and R. A. Athale, “Active-eyes: an adaptive pixel-by-pixel image segmentation sensor architecture for high-dynamic-range hyperspectral imaging,” Appl. Opt. 41, 6093–6103 (2002).

3. E. Clarkson and H.H. Barrett, “Approximations to ideal-observer performance on signal detection tasks,” Appl. Opt. 39, 1783-1793 (2000).

4. W.C. Chou, M. A. Neifeld, and R. Xuan, “Information-based optical design for binary-valued imagery,” Appl. Opt. 39, 1731-1742 (2000).

5. R.D. Swift, R.B. Wattson, J.A. Decker, R. Paganetti, and M. Harwit, “Hadamard transform imager and imaging spectrometer,” Appl. Opt. 15, 1595-1609 (1976).

6. Mark A. Neifeld and Premchandra Shankar, “Feature-specific imaging,” Appl. Opt. 42, 3379-3389 (2003). 7. M. Antonini, M. Barlaud, P. Mathieu, and I. Daubechies, “Image coding using wavelet transform,” IEEE

Trans. Image. Process. 1, 205-220, (1992). 8. S. J. Lee, S. B. Jung, J. K. Kwon, and S. H. Hong, “Face detection and recognition using PCA,” TENCON

99, Proc. IEEE Region Conf. 1, 84-87 (1999). 9. R. Stephen and E. Richard, Independent Component Analysis: Principles and Practice, Cambridge

University Press, Cambridge, UK, 2001, Chap. 1, 1-70. 10. C. J. Oliver, “Optical image processing by multiplex coding,” Appl. Opt., 15, 93-106 (1976). 11. K. Hsu, H. Li, and D. Psaltis, “Holographic implementation of a fully-connected neural network,” Proc.

IEEE, 78, 1637-1645 (1990). 12. E. Marszalec, B. Martinkauppi, M. Soriano and M. Pietikäinen, “Physics-based face database for color

research,” J. Electron. Imaging 9, 1, 32-38, 2000. 13. Earth science data interface, Global Land Cover Facility, University of Maryland. 14. D. S. Taubman and M. W. Marcellin, JPEG2000: Image Compression Fundamentals, Standards, and

Practice (Kluwer, Boston, MA, 2002) Chapter 4, 157-158.

1. Introduction

Imaging is a primary step in many modern applications including surveillance, astronomy, satellite remote sensing, medical diagnosis and others. Almost all imaging applications today

(C) 2003 OSA 8 September 2003 / Vol. 11, No. 18 / OPTICS EXPRESS 2118#2708 - $15.00 US Received July 09, 2003; Revised August 18, 2003

mailto:[email protected]

mailto:[email protected]

strive to generate a visually pleasing image, even though the eventual system performance metric may be quite different. This suggests that overall system performance may be improved by optimizing the optical image acquisition hardware based on the specific criteria required by each application [1-4]. This observation has given rise to the development of several non-traditional imaging systems that capture features which are optimal according to task-dependant system design metrics [5,6]. For example, if an imager is required to transmit its imagery, then wavelet features might be selected to optimize compression efficiency [7]. Alternately, Karhunen–Loeve features may be used to minimize the root mean square error (RMSE) within a pattern recognition application, or independent component features may be used to maximize the information extracted from the images [8,9].

We recently presented a candidate system for feature-specific imaging that directly measures linear features in the optical domain [6]. It was demonstrated that the multiplex advantage of the system could provide improved performance as compared with features obtained by post-processing conventional images. Studies on principal component (PCA) features revealed that even reconstruction RMSE could be superior to conventional imaging within high noise environments. Many imaging applications such as satellite imaging of earth or deep space are multispectral, and in such applications, spectral correlation may be exploited to further improve imager performance. In this paper, we extend the work of reference [6]; we analyze the direct optical measurement of multispectral PCA features and compare the resulting reconstruction quality with the performance of a conventional imager.

2. Feature specific imaging

In this section, we briefly review the basic idea of feature-specific imaging. Because this paper is a multispectral extension to the work reported in reference [6], we limit the scope of this review and refer the reader to that earlier work for additional details. A candidate optical system for direct optical feature-extraction was presented in reference [6]. We will not limit the analysis presented here to that specific optical system. Instead, we present an analysis that is relevant to any optical system that seeks to directly measure linear projections of an object irradiance distribution. Many such systems have been previously described in the literature, including approaches that exploit absorptive, diffractive, and/or polarization-based optical processing [10,11,6].

PCA feature-specific imaging begins with a training phase. During this phase, we must train the imager on a database of representative images in order to generate the principal component basis vectors. Consider an imager that measures k spectral components. Each training image is treated as a set of √N×√N×k pixel blocks. Overlapping blocks are used during training in order to ensure shift-invariance during imager operation. We find that for the imagery used in this work, an overlap of 75% or greater is sufficient for this purpose. The blocks are treated as vectors and their covariance matrix (H) is computed. This process exploits both the spatial and spectral correlation structure of the training imagery. The eigenvectors of H (i.e., the principal components) define a set of basis vectors onto which a PCA feature-specific imager will project its incident object irradiance distribution. If the object is similar to the training images, relatively few features are sufficient to capture most of the object variance: a defining characteristic of principal component analysis.

Consider an imager that measures M features (1 ≤ M ≤ kN). We can arrange the M basis vectors that define the projections for such an imager, as the rows of a projection matrix W. For PCA imaging these M rows will correspond to the eigenvectors associated with the M largest eigenvalues of H. The object, assumed spatially discrete, is treated as a set of non-overlapping √N×√N×k pixel blocks. Blockwise processing of the object irradiance distribution facilitates measurement of local features. Features are optically computed on individual object blocks, each block being treated as a column vector x, of dimension kN. The M-dimensional measurement vector m is a corrupted version of the true feature vector y = Wx; mathematically m = Wx + n, where W is the M x kN dimensional projection matrix and n is additive white Gaussian noise (AWGN) originating in the detector and/or detection electronics. The noise standard deviation is given by σ.


Any optical implementation of the projection operation Wx is subject to two physical constraints. First is a photon count constraint. Note that xi (the ith element of x) represents the number of photons that originate at the ith object pixel and are collected by the imager. In order to insure a fair comparison between conventional and feature-specific imaging we require that the projection operation can only re-distribute these available photons among the M features: no photons can be created within the imager. The ith column of W specifies how this re-distribution should occur for xi with i = 1, … kN. The photon count constraint therefore requires that the absolute sum of each column of W be upper-bounded by one. For this reason, we must scale the W obtained from the eigenvectors of H, by its maximum absolute column sum C.

The second constraint for optical implementation is the need for dual-rail signaling. Because the negative values of the projection matrix W cannot be implemented directly, two complementary arms are required. A schematic representation of such an imager operating with three spectral components (k=3) is shown in Fig. 1. Here, one arm uses W+ to generate the measurement m+ = tW+x/C+ + n, and the other arm uses W– to compute m– = (1–t)W–x/C– + n. The elements of W+ are given by w+(i,j) = w(i,j) if w(i,j) >0, else w+(i,j) = 0, and those for W– are given by w–(i,j) = –w(i,j) if w(i,j) <0, else w–(i,j) = 0. C+ and C– are the maximum column sums of W+ and W– respectively, and t is the power-splitting ratio of the dual-rail system. Because C+ and C– are not necessarily equal, the optimal value of t is not 0.5 but can be determined by minimizing the feature RMSE to yield t = (C+)2/3/[(C+)2/3 + (C–)2/3]. The final multiplexed features ym are obtained by subtracting the scaled output of the negative arm from that of the positive arm, ym = C+m+/t – C–m–/(1-t). Any such implementation will require two detectors per block to measure each feature.

Due to the photon count constraint, most columns of the scaled W+ and W– sum to less than one, which decreases photon detection efficiency. An alternate architecture for the multispectral multiplexed imager can be designed, in which each of the k spectral components is measured in a separate arm. This approach will require 2k arms, each scaled independently, and will result in a slight improvement in photon detection efficiency; however, the number of detectors has increased k-fold. This causes the overall system measurement SNR to go down approximately k-times, and negates any small advantage achieved by independent scaling of the spectral components. Our simulations have shown that the performance of such an approach is consistently inferior to the architecture shown in Fig. 1, and all results presented throughout the remainder of this paper are therefore based on the 2-arm approach.

Fig. 1. Schematic diagram of a multispectral feature-specific imager with three (RGB) bands


3. Feature fidelity comparison

Feature fidelity is quantified by use of the root mean square error (RMSE) of the measured features. The system in Fig. 1 produces a feature measurement with RMSEm = {<|y – ym|

2>/M}0.5 which reduces to RMSEm = [{C+/t}2 + {C–/(1-t)}2]0.5σ, where < > is the statistical expectation operator. It can be easily shown that the feature RMSE for conventional imaging is given by RMSEc = σ. In feature-specific imaging, the noise is scaled by a factor that depends on C+ and C–. A multiplex advantage will be achieved if the number of rows in W is kept small so that the column sums C+ and C– are also small producing {C+/t}2 + {C–/(1-t)}2 < 1. The feature-specific imager will be superior to conventional imaging when this condition is satisfied. In the proposed imager architecture, only two detectors are required to measure a feature. Because M << kN, the average number of photons detected by each sensor is larger than in the case of conventional imaging. This integration across spatial and spectral components thus generates higher measurement SNR in feature-specific imaging as compared with conventional imaging.

The above analysis of feature RMSE is completely general. Because the expression for RMSEm involves the quantities C+ and C–, and because these constants depend upon the specific projection matrix under consideration, no further progress can be made in the general case. In order to provide additional insight into this comparison we use examples from two different types of multispectral images. We use color face images (3 spectral components, RGB), and LANDSAT-7 images (7 spectral components) [12,13]. Figure 2(a) shows some images from the RGB face image database. 100 RGB face images are used in our simulations, 99 images for training the PCA projection matrix, and 1 image for testing. The testing image is not part of the training set. We use a leave-one-out training strategy and average the results over all 100 such training/testing sets. Figure 2(b) shows the 7 components of a single LANDSAT-7 image; 35 such images are used for training and 1 for testing. Once again, we employ a leave-one-out training strategy in which the testing image is never part of the training set. We first present results for the face images. The PCA features are computed for various block sizes and the resulting feature RMSE is plotted in Fig. 3(a) as a function of the number of measured features (M). The RMSE is expressed as a percentage of 255, which is the maximum pixel value. This figure compares the feature fidelity of multispectral (MS) and grayscale (GS) feature-specific imaging with that of conventional imaging for 4×4 (kN=3×4×4=48 pixel), 8×8 (kN=192 pixel) and 16x16 (kN = 768 pixel) block sizes. We observe that the feature fidelity of both MS and GS feature-specific imaging is superior to that of conventional imaging when the number of features is small. For example, with 16x16 blocks and a noise level of σ = 20 we find that the RMSE of the optically measured features

(a)

(b)

Fig. 2. Example of multispectral images. (a) Face images [3 spectral components]. (b) A LANDSAT-7 image [7 spectral components and their wavelength ranges in microns].


will be smaller than that of features computed from a conventional image for M<10 and M<6 in the MS and GS cases respectively. We also note that the feature RMSE of the RGB imager is lower than that of the grayscale imager for all M. This is because for every M the maximum column sum C is smaller for the RGB imager owing to the variance of the object irradiance being distributed among a larger set of basis vectors. Thus, the feature fidelity increases due to improved photon detection efficiency and lower noise scaling. For the same reason, both C and feature RMSE decrease with increasing block size from 4×4 to 16x16. It can be seen that as M increases, the number of rows in W increases causing increased C, and hence a rise in the feature RMSE. Similar results are plotted in Fig. 3(b) for the LANDSAT-7 images with 4x4 blocks (kN=112 pixels), 8x8 blocks (kN=448 pixels), and 16x16 blocks (kN=1792 pixels) and similar trends are observed. For the same block size and M, the Landst-7 imager captures a smaller fraction of all possible features, and so has a lower C and higher feature fidelity as compared with the RGB imager.

0

4

8

12

16

20

0 2 4 6 8 10 12Number of Features [M]

fea

ture

RM

SE

[%

]

conventionalGS 4x4MS 4x4GS 8x8MS 8x8GS 16x16MS 16x16

0

4

8

12

16

20

0 2 4 6 8 10 12Number of features [M]

Featu

re R

MS

E [%

]

ConventionalGS 4x4MS 4x4GS 8x8MS 8x8GS 16x16MS 16x16

(a) (b)

Fig. 3. Feature fidelity (RMSE) versus number of measured features (M) for multispectral (MS), grayscale (GS), and conventional imaging, with σ=20. (a) Data obtained using face (RGB) images. (b) Data obtained using LANDSAT-7 images.

4. Reconstruction fidelity comparison

In the previous section, we found that a tradeoff exists between the number of features captured and the feature fidelity. This tradeoff arises from the photon count constraint. In this section, we present simulation results on reconstruction RMSE, for which (noise-free) PCA features are optimal. The reconstruction RMSE for conventional imaging is simply due to the detector noise, and hence RMSEc = σ. However, for a multiplexed imager capturing M out of kN total PCA features, the reconstruction RMSE of each pixel is given by

−+

+=

−+

+=∑

22

2

1 1

1

t

C

t

CM

kNRMSE

N

Miim σλ

where λi is the ith largest eigenvalue of the PCA covariance matrix. Figure 4 shows the plot of reconstruction RMSE vs. M at σ = 30, and shows the expected tradeoff in reconstruction RMSE. It is observed that for 16x16 blocks, if the number of features is below 15, the number of detectors is small and so the effect of detector noise is negligible. The RMSE is dominated by error due to truncating (kN-M) features. This error is minimized by leaving out the projections with minimum variance, and decreases with increasing M. However, detector


noise starts to dominate as M increases due to increasing C. A minimum is obtained when these two effects are balanced. The data in Fig. 4 shows that this optimum imager design can provide significant RMSE improvement as compared with conventional imaging. In particular, the feature-specific imager is seen to provide superior performance with 16x16 blocks measuring fewer than 50 features. The optima can be extracted from Fig. 4 and plotted as a function of noise. The resulting plot of minimum reconstruction RMSE vs. σ is given in Fig. 5(a). We see that in high noise environments, the reconstruction fidelity of feature-specific imaging can be better than that of conventional imaging. The dependence on the block size is not monotonic, and the optimal block size depends on the type of images for which the imager is used. The use of 6x6 blocks is optimal for these face images. The green curve in Fig. 5(a) demonstrates that if the test image is not similar to the training images, the optimal reconstruction RMSEm will increase. An analogous plot for the LANDSAT-7 data is shown in Fig. 5(b), and has similar characteristics. The reconstruction RMSE in the LANDSAT-7 imager is comparatively higher than that of the RGB imager, due to the difference in the PCA projection matrix in the two cases. The projection matrix depends on the spatial and spectral correlation within each block, which differs with the type of images used. At a fixed σ, the minimum reconstruction RMSE for the LANDSAT-7 imager is obtained at higher M, which also accounts for its lower fidelity. An original “face” test image along with example reconstructions from both feature-specific and conventional imaging are

0

5

10

15

20

25

30

0 10 20 30 40 50 60Number of features [M]

reco

nst

ruct

ion

RM

SE

[%

]

conventional imaging8x8 blocks16x16 blocks

Minimum reconstruction

Crossover

Fig. 4. Reconstruction fidelity (RMSE) versus number of measured features (M) for multispectral feature-specific imaging and conventional imaging. Data obtained using face (RGB) images with σ=30.

0

2

4

6

8

0 4 8 12 16 20Receiver Noise Standard Deviation

Min

imim

Recon

str

uctio

n R

MS

E [

%]

Image different from training setconventional imagingMS 4x4 blocksMS 6x6 blocksMS 8x8 blocksMS 16x16 blocks

0

4

8

12

16

20

0 10 20 30 40 50Receiver noise standard deviation

Min

imu

m r

eco

nstr

uctio

n R

MS

E [

%]

Conventional imagingMS 4x4 blocksMS 6x6 blocksMS 8x8 blocksMS 16x16 blocks

(a) (b)

Fig. 5. Minimum reconstruction RMSE versus noise for MS images. (a) Results with Face images. (b) Results with LANDSAT-7 images.


Original 2x2 blocks 4x4 block 6x6 blocks 8x8 blocks Conventional No noise 2 features 4 features 5 features 8 features imaging RMSE 4.6% RMSE 4.3% RMSE 3.5% RMSE 3.7% RMSE 5.5%

Fig. 6. Example reconstructions of a face image using several different block sizes. σ=14 in these examples.

shown in Fig. 6. It is interesting to note that even in those cases for which visual quality is the appropriate performance metric, feature-specific imaging may produce better results than conventional imaging. This observation is further supported by the example LANDSAT-7 images presented in Fig. 7, which are example reconstructions of the image in Fig. 2(b). Figure 7(a) shows the 7 bands of a noisy LANDSAT-7 image obtained via conventional imaging with σ = 40. The resulting image fidelity is given by RMSEc = 15.7%. Figure 7(b) depicts the results of PCA feature-specific imaging using M=21. The visual quality improvement is confirmed by the significantly lower RMSEm = 7.6%.

(a)

(b)

Fig. 7. Example reconstructions of a LANDSAT-7 image at σ=40. (a) Reconstruction from conventional imager. (b) Reconstruction from multiplexed imager with 16x16 blocks.

5. Conclusion

This paper presents a study of feature-specific imaging in the multispectral domain. We have shown that a multispectral feature-specific imager can achieve better fidelity (i.e., lower RMSE) than conventional imaging when measuring linear features from the collected object irradiance distribution. Although this study was carried out using PCA features, it can be easily generalized to other linear features e.g. wavelet, Hadamard, or independent component features [6]. We have quantified the tradeoff between the number of features and feature fidelity, and we have shown that the multiplexed advantage causes the multispectral imager to achieve higher feature fidelity than its grayscale counterpart. We also present the somewhat surprising result that reconstruction fidelity can also benefit from the multiplex advantage. This is demonstrated by significantly lower reconstruction RMSE as compared to conventional imaging in high noise environments. In addition to these fidelity gains, feature-specific imaging also results in reduced detector count, which lowers both imager hardware complexity and data rate.


It is interesting to note that in the presence of both measurement noise and the photon count constraint, PCA is no longer optimal in terms of reconstruction fidelity. The search for optimal projections that might be used in feature-specific imaging is an area of continuing research. In general, the use of image-dependent basis vectors has both positive and negative implications. One advantage is that the resulting imagers are truly task-specific. This means that it may be possible to guarantee the best possible performance using such an imager. One disadvantage is the resulting dependencies on training set size and the required degree of similarity between training and testing data. Previous theoretical work in this area has shown that as the size and diversity of the training set grows, PCA approaches the discrete-cosine-transform [14]. In the case of feature-specific imaging we have not studied these dependencies in detail; however, we do consider such a study to be an important area of future work.


Documents

Multispectral principal component imaging