IMAGE FUSION PERFORMANCE ANALYSIS BY ANOVA · image fusion process is to create a good multiresolution decomposition by choosing its adequate variables (wavelet type and number of

IMAGE FUSION PERFORMANCE ANALYSIS BY ANOVA

NATALIA R. DOS SANTOS

Divisão de Engenharia Mecânica, Departamento de Mecatrônica, Instituto Tecnológico de Aeronáutica,

12228-900 São José dos Campos, SP, Brasil

E-mails: [email protected]

ELDER M. HEMERLY

Divisão de Engenharia Eletrônica, Departamento de Sistemas e Controle, Instituto Tecnológico de Aeronáutica,

12228-900 São José dos Campos, SP, Brasil

E-mails:[email protected]

Abstract Image fusion techniques play an important role in several fields of knowledge, due to its ability to concentrate re-

dundant and complementary information, without distorting them, into one single image, which is more appropriate for human or

machine processing. There are many ways to perform the image fusion procedure, and several variables to be considered in their

design. The goal of this work is to present a systematic performance analysis of image fusion methods, with statistical justifica-tion. A consistent method, based on ANOVA (analysis of variance) concepts, is employed to compare results from different in-

puts of the fusion procedure, by using objective performance criteria for the fused image.

Keywords image fusion, wavelet transform, ANOVA, image fusion performance criteria, multi scale decomposition.

Resumo Técnicas de fusão de imagem cumprem um papel importante em diversas áreas do conhecimento, devido à premissa

de concentrarem informações redundantes ou complementares de duas ou mais fontes, sem que haja distorção, a fim de produzir

uma imagem única que seja mais apropriada ao processamento por seres humanos ou máquinas. Há muitas maneiras de se efetuar fusão de imagens, e diversas variáveis a serem consideradas no projeto. O objetivo deste trabalho está em sistematizar a análise

dos resultados de fusões, usando justificativa estatística, ao definir um método consistente baseado nos conceitos da ANOVA

(análise de variância), para compará-los quando gerados por diferentes variáveis de projeto de fusão, usando critérios objetivos de desempenho para a imagem final.

Palavras-chave fusão de imagens, transformada wavelet, ANOVA, critérios de desempenho para fusão de imagens, decompo-

sição em múltiplas escalas.

1 Introduction

Image fusion methods have been consistently

used in several applications, such as in geology

(disaster monitoring), medicine (PET/CT machines),

remote sensing (satellite imaging), military (target

recognition), safety &security (law enforcement,

surveillance), industrial (quality inspection) (Li, et

al., 1994) (Cheng, et al., 2008) (Yang & Blum,

2002). This wide range of applicability has attracted

much interest to this area, and several researches

have been conducted in the last decades, aimed at

finding techniques to improve the fusion results

(Choi, et al., 2010).

The main purpose of the image fusion techniques

is to produce a single image, using complementary

or redundant information from two or more sources,

which is more suitable for human or machine per-

ception and treatment (Zhang & Blum, 1999). The

expected results are usually the improvement in the

quantity and/or quality of information and simulta-

neously the decrease in the amount of data to be

analyzed (Petrovié & Xydeas, 2004), manually or

automatically. The images to be used in the fusion

processing may come from different sensors (multi-

sensor), or from the same sensor (multifocus appli-

cations for instance), but containing different fea-

tures. Practical examples are shown in Figure 1-3.

Figure 1: Example of medical application. Primary tumor is not well seen on noncontrast CT (A) but is clearly delineated on PET

(B) and PET/CT fusion (C) images (JNM, 2014)

Figure 2: Example of military application in target recognition:

a) IR sensor showing physical edges; b) EO sensor-objects cov-

ered by smoke and c) Fused Image – physical features and smoke represented. (SPIE, 2014)

Anais do XX Congresso Brasileiro de Automática Belo Horizonte, MG, 20 a 24 de Setembro de 2014

181

Figure 3: Multifocus source images fusion result. a) focus on left;

b) focus on right and c) fused image.

The Discrete Wavelet Transform (DWT) is a

suitable tool for the image fusion application and is

the most used technique in recent works (Amolins,

et al., 2007). The procedure involves decomposing

the source images in their multiresolution wavelet

representations, and then manipulating the obtained

data to generate the final image. The key to any

image fusion process is to create a good

multiresolution decomposition by choosing its

adequate variables (wavelet type and number of

decomposition levels) and then defining a fusion

rule that performs well. See Figure 4 for details.

Figure 4: Basic scheme for DWT image fusion.

Results are usually registered in tables and the

assessment of the data is commonly done by visual

comparison of the results (Yang, 2011) (Zhang &

Blum, 1999) (Gonzalo & de la Cruz, 2004) (Zeng, et

al., 2012) (Luo, et al., 2012) or by a simple graphical

representation without any further mathematical

background. In several other fields of knowledge,

this background is a must and the systematization of

analysis results using statistical justification is the

default procedure, especially when dealing with

biological samples. This can be seen in many medi-

cal studies (Koehler, et al., 2014).

This paper introduces the statistical parameteri-

zation on the data analysis. An experimental

procedure for evaluating the performance of image

fusion methods is proposed, by considering several

combinations of wavelet types, number of decompo-

sition levels, source images and fusion rule methods.

This data analysis relies on the computational im-

plementation of the well-known ANOVA

(Montgomery, 2001) concepts for performing the

required analysis. The main goal of this paper is

then to define a consistent method for analyzing

image fusion results. An automated statistical ap-

proach is used for generating and evaluating the

outputs produced by several changes in input pa-

rameters of a DWT based fusion. This approach

allows quick and reliable selection of methods and

parameters for the desired fusion application, and

permits incremental changes to cope with new de-

velopments. This flexibility is due to the fact that

new performance criteria and processing methods

themselves can be adapted as variables to the analy-

sis. This way a bigger set of data can be analyzed at

once and the results statistically compared and justi-

fied.

2. Image fusion and statistical analysis

2.1 Wavelet based image fusion

The DWT is nowadays the most used tool for

applications in image fusion and image compres-

sion. Images are two dimension objects, computa-

tionally represented by matrices where each element

defines a pixel whose properties are associated to

this element value.

The basic principle is to perform the signal

analysis by decomposing it into its frequency com-

ponents, like it is done when transformed methods

are used, such as the Fourier Transform. The main

difference lies on the reference used (Gonzalo & de

la Cruz, 2004). In contrast to the Fourier methods,

where the basic functions are sinusoids, the basic

element are now small waves (the wavelets), whose

main characteristics are variable frequency, finite

duration, mean value equal to zero and usually

assymetry.

When compared to the usual Fourier transform,

the wavelet transform has a striking advantage: it is

capable of mapping both the frequency and temporal

position of the original signal, so that every resolu-

tion is tied to its scale. In short, the temporal infor-

mation is not lost in the transformation process.

This approach has notable advantages for signals

that contain several peaks and discontinuities, which

is the case of images. An illustration for this proper-

ty is provided in Figure 5.

Figure 5: Time-frequency/scale representation of the Wavelet

Transform. (MATLAB, R2010A)


182

The use of the Wavelet Transform for image

processing is based on the MRA methods (Multi

Resolution Analysis), technique in which the signal

is decomposed into different resolutions. This ap-

proach follows the principle that distinct elements

have their representation more suited to a specific

resolution level. The level represents a certain reso-

lution and the decomposition creates 4 new groups

on every level, being one for the approximation (low

frequency) coefficients and another 3 for the detail

(high frequency) coefficients, but each one operates

on a different direction (horizontal, vertical and

diagonal) (Gonzalez & Woods, 2002). See Figure 6.

Figure 6: Multiresolution decomposition structure for a 3 level decomposition.

The high frequency bands of an image provide

the details of the scene. The corresponding coeffi-

cients depict strong intensity variations and preserve

salient information on the images, and are character-

ized by high absolute values. On the other hand, the

low frequency bands of an image supply the basic

and coarse information of the depicted scene. The

corresponding coefficients representation is equiva-

lent to a low resolution view of that image and gen-

erally has small absolute values, being usually re-

ferred to as “approximation” coefficients.

The fusion method using DWT relies on three

main steps: 1) Decompose the source images into

their multi resolution scales, generating a matrix of

coefficients for each of the sources. This is done by

filtering and down sampling the initial data. This

first step supplies the high and low frequency coef-

ficients mentioned before. 2) The second step is the

subject which attracts the largest attention in the

literature: it deals with the rules for actually creating

a composite image, by using algorithms to merge

or choose the coefficients from each image source

and 3) The new and single coefficients matrix that

was created goes through and inverse wavelet trans-

form, that finally generates the fused image.

2.2 ANOVA Analysis

According to statistics premises, one result is

only different from another if, after analyzing the

variance of the samples, considering or not the in-

fluence factors as random effects, the difference

between the mean values is within a pre-set proba-

bility of happening, which is called statistically

significant. (Montgomery, 2001).

3 Methods

Tests were done changing the MSD variables

(wavelet family and number of levels) and the fu-

sion method. Grey level images were used, so as to

simplify the experiment. The chosen performance

metrics were RMSE (root mean square error), entro-

py and overall cross entropy. See Table 1 for details.

Table 1: Variables tested on the experiment

Factors Levels Values

Sample 1 Cameraman.tiff

Wavelet 5 'db8', 'db2', 'haar', 'sym2',

'sym6'

Fusion

Rule

2 Joint (mean-mean),

Individual (Yang, 2011)

Number

of levels

4 2, 3, 5, 8

In this work the fusion procedure was imple-

mented in MATLAB (MATLAB, R2010) and the

core function used was “wfusimg”, from the image

processing package. This function uses as variables

all the factors that were exercised in this experiment.

Two fusion rules were investigated: 1) a joint rule

for all wavelet decomposition coefficients , based on

averaging them, here called mean-mean method,

which is very simple and intuitive, and 2) frequency

band individualized rules as proposed by (Yang,

2011), which is quite promising, since it is able to

capture relevant information from different signal

frequency ranges.

The mean-mean rule is a method where the co-

efficients of both MSD matrixes are averaged in

order to compose the fused matrix.

The rule proposed by (Yang, 2011) considers

high and low frequency information given by (1)

and (2) respectively

( ) { ( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )

where Dx and Dy are coefficients on the multiscale

representation for the source images and Dz for the

fused image. Dz is chosen based on the maximum

variance in a window for each pixel in (1) and is

selected by Wx and Wy in (2) whose values can be

1or 0 according to maximum edge detection rule.


183

3.1 The source image

Multifocus source images refer to the images

taken from a unique scene using the same sensor,

but with different quality in each portion of the

image, due to changes in the focus. This is equiva-

lent to what happens when shots are taken by a cam-

era, with focus in two parts of the image. On the

other hand, multisensor images refer to a scene

captured by different sensors such as infra-red, elec-

tro-optical, magnetic resonance, etc.

In multifocus applications, it is possible to con-

catenate the portions of the source images whose

focus is adequate and generate a final image with

desired quality. This is an artisanal process though,

which can be easily simulated by blurring different

parts of a single image. The advantage of using a

single reference image lies on the fact that it pro-

vides a “control group”, also called ground truth,

which simplifies performance evaluation.

In this work input images were created by blur-

ring the right and left sides of the original image, by

using a symmetric Gaussian low pass filter. Two

well know images were used: the Lena and Camer-

aman, both of size 256 x 256, gray level on TIFF

format.

3.2 Performance Criteria

In the past the results of image fusion were

commonly evaluated by visual assessment (Li, et al.,

1994). The quantitative evaluation of performance is

a complicated issue due to the fact that the ideal

result is usually unknown (Zhang & Blum, 1999),

especially when working with multisensor image

fusion which appears in most real applications

(Zheng & Qin, 2009). Multifocus applications and

its simulations, as done in this work, can be better

evaluated because it allows comparison to the

ground-truth reference. Performance criteria are also

classified according to these reference requirements.

Three complementary performance indices are used

here: RMSE, Entropy and Overall Cross Entropy

(OCE) shown in Equations 3-6.

(

∑ ∑( ( ) ( ))

)

( )

where xR is the reference image (ground-truth),

xF is the fused image and M and N represent its size.

The lower the RMSE value, the better the fusion.

∑ ( )

where P is the probability of a particular gray

level occurrence on that image, obtained from its

histogram, and L the number of gray levels. The

Entropy represents an index for reflecting the

amount of information in the image, which is statis-

tically measured by the randomness of the infor-

mation. The higher, the better.

( ) ( )

( )

∑

( )

where P and Q are histograms of source and fused

image. A,B and F are the source and fused images,

respectively. This index is a measure for differences

to the final image, therefore, the smaller, the better.

RMSE is a good performance indicator for stud-

ies with simulated databases because it takes the

advantage of using the reference image, being a very

straight way of comparing them. The entropy and

overall cross entropy methods are based on the

physical meaning of the image, thus, not requiring a

reference. The drawback is that they are susceptible

to particularities of each sample, such as amount of

details on image.

4. Results

The plots in Figure 7-9 show the mean error values

for the three performance methods chosen in section

3.2.

Figure 7:RMSE versus each tested factor

2.0

2.5

3.0

3.5

4.0

Means of Factors x RMSE

Factors

RM

SE

db2 db8

haar

sym2 sym6

individual

joint

2

3 5 8

Wavelet Fusion_Rule Nb_Levels


184

Figure 8: Overall Cross Entropy versus each tested factor

Figure 9: Entropy versus each tested factor

The means showing divergent results for each

performance index indicate that they actually meas-

ure different things. This is an important issue that

expresses the importance of adequately choosing a

performance criterion to the desired application of

the fused image.

The ANOVA also showed different results for

each performance metric as seen in Table 2. Howev-

er, some factors are clearly significant in all three

analyses: the fusion rule and wavelet type, as ex-

pected. These results are also very useful to under-

stand the behavior of interactions among the factors,

which informs the effects of a certain parameter

beyond its own influence and ultimately indicates

how difficult it is to optimize the response under the

parameter variation.

It should be stressed, however, that the main

goal here is not to select the best method, but actual-

ly how to compare performance of different meth-

ods, with statistical significance.

Table 2

Factors RMSE Entropy OCE

Wavelet *** *** ***

Fusion Rule *** *** ***

# Levels *** ** ***

Fus Rule : #Levels *** * ***

Wavelet : #Levels *** ** x

Wavelet : Fus Rule x ** ***

Model Adequacy 99.9% 98.7% 97.7%

Significance codes:

0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘x ’ 1

The evaluation of the ANOVA premises of in-

dependency, normality of distribution and homosce-

dasticity of the residuals indicate that the analysis is

valid, since none of them is considered to be violat-

ed, according to the graphical evaluations shown in

Figure 10.

Figure 10: ANOVA premises analysis

A closer analysis of the RMSE results, by look-

ing at the interaction plots in Figure 11, indicates

considerably different results for each fusion rule as

the number of levels increases. There is no effect for

the averaging rule and a clear tendency of improving

the results until 5 levels of decomposition for the

individual fusion rule. This graphical analysis also

leads to the conclusion that the significant response

of the levels found in ANOVA is only present in

individual fusion rule, which finally indicates an

interaction between these two factors (fusion rule

and number of levels).

7.0

1

7.0

2

7.0

3

7.0

4

Means of Factors x Entropy

Factors

En

tro

py

db2

db8 haar

sym2

sym6

individual

joint

2 3

5 8


7.8

84

7.8

86

7.8

88

7.8

90

Means of Factors x Overall Cross Entropy

Factors

Ove

rall

Cro

ss E

ntr

op

y

db2

db8

haar

sym2

sym6

individual

joint

2

3

5 8


0 10 20 30 40

-0.0

6

0.0

2

Independency

Order

Resid

ua

ls

-2 -1 0 1 2

-0.0

6

0.0

2

q-q plot-Normality

Predicted

Resid

ua

ls

Histogram-Normality

Residuals

Fre

que

ncy

-0.05 0.00 0.05

0

4

8

12

1.5 2.5 3.5

-0.0

6

0.0

2

Homoscedasticity

Fitted Residuals

Resid

ua

ls


185

Figure 11: Interaction plot of number of levels and Fusion rule

5. Conclusion

A statistical procedure was proposed in this

work for comparing performance of image fusion

methods. The ANOVA technique was implemented

and showed consistent results: it supplies technical

and mathematical backgrounds for the performance

analysis of fused images. The influence of several

design parameters can be investigated, thus reducing

considerably the amount of data necessary to obtain

conclusive and practical results, with statistical

meaning.

6. References

Amolins, K., Zhang, Y. & Dare, P., 2007. Wavelet

based image fusion techniques-An

introduction,review and comparison. ISPRS

Journal of Photogrammetry & Remote Sensing.

Cheng, S., He, J. & Lv, Z., 2008. Medical Image of

PET/CT Weighted Fusion Based on Wavelet

Transform. IEEE 978-1-4244.

Choi, K., Kim, C. & Beon Ra, J., 2010. Infrared

Image enhancement based on aligned high

resolution visible. IEEE-ICIP.

Gonzalez, R. C. & Woods, R. E., 2002. Digital

Image Processing. Second ed. New Jersey:

Prentice-Hall Inc.

Gonzalo, P. & de la Cruz, J. M., 2004. A wavelet-

based image fusion tutorial. 0031-3203 Patttern

Recognition Society.

JNM, T. j. o. N. M., 2014. jnm.snmjournals.org.

[Online]

Available at:

http://jnm.snmjournals.org/content/47/5/755.fig

ures-only

[Accessed 20 March 2014].

Koehler, G. et al., 2014. Insulin degludec is not

associated with a delayed or diminished

response to hypoglycaemia compared with

insulin glargine in type 1 diabetes: a double-

blind randomised crossover study.

Diabetologia, pp. 40-49.

Lehigh University, B., 2014. ece.lehigh.edu.

[Online]

Available at:

http://www.ece.lehigh.edu/SPCRL/IF/image_fu

sion.htm


Li, H., Manjunat, B. & Mitra, S., 1994. MULTI-

SENSOR IMAGE FUSION USING THE

WAVELET TRANSFORM. IEEE.

Luo, B. et al., 2012. Decision-based fusion for

pasharpening of remote sensing images. IEEE

Explore.

Mallat, S. G., 1989. A Theory for Multiresolution

Signal Decomposition: The Wavelet

Representation. IEEE Transactions on Pattern

Analysis and Machine intelligence.

Montgomery, D. C., 2001. DESIGN AND

ANALYSIS OF EXPERIMENTS. Fifth Edition

ed. s.l.:s.n.

Petrovié, V. S. & Xydeas, C. S., 2004. Gradient-

Based Multiresolution Image Fusion. IEEE

Transactions on Image Processing.

Soliman, R., Amin, M. & Abd El-Samie, F., 2012.

Enhanced Fusion for Infrared and Visible

Images. Cairo, s.n.

SPIE, I. S. f. O. a. P., 2014. spie.org. [Online]

Available at: http://spie.org/x32828.xml


Yang, J. & Blum, R., 2002. A stattistical signal

processing approach to image fusion for

concealed weapon detection. IEEE ICIP.

Yang, Y., 2011. Multiresolution Image Fusion

Based on Wavelet Transform by Using a Novel

technique for Selection Coefficients. Journal of

Multimedia.

Zeng, Y., van Genderen, J., Zhang, J. & Wang, G.,

2012. A22Evaluation Criteria for Image Fusion

Performance in Different Applications. 2012

International Conference on Industrial Control

and Electronics Engineerig.

Zhang, Z. & Blum, R. S., 1999. A Categorization of

Multiscale-Decomposition-Based Image Fusion

Schemes with a Performance Study for a Digital

Camera Application. PROCEEDINGS OF THE

IEEE, August.

Zheng, Y. & Qin, Z., 2009. Objective image fusion

quality evaluation using structural similarity.

Tsighua science and te chnology.

1.5

2

.0

2.5

3

.0

3.5

4

.0

Factors Interaction Plot

Number of Levels

RM

SE

2 3 5 8

Fusion_Rule

joint individual


186

Documents

IMAGE FUSION PERFORMANCE ANALYSIS BY ANOVA · image fusion process is to create a good multiresolution decomposition by choosing its adequate variables (wavelet type and number of