Effects of multitemporal scene changes on pansharpening fusion

Effects of Multitemporal Scene Changes on Pansharpening Fusion

Trento, Italy, 12-14 July 2011Multitemp 2011

B. Aiazzi*, L. Alparone°, S. Baronti*, R. Carlà* , A. Garzelli^, L. Santurri* , M. Selva*

*Institute of Applied Physics “Nello Carrara”,Research Area of Florence, Sesto Fiorentino, Italy°Department of Electronics & Telecommunications, University of Florence, Florence, Italy

^Department of Information Engineering, University of Siena, Siena, Italy

Outline

• Pansharpening achieved as a component substitution (CS).

• Pansharpening achieved through multiresolution analysis(MRA).

• Behavior of CS and MRA pansharpening methods in the presence of temporal scene changes between multispectral (MS) and panchromatic (PAN) observations.


(PAN) observations.

• Results on very high resolution (VHR) GeoEye-1 data.

• Concluding remarks

Pansharpening Fusion Issues

Detail Extraction Detail Injection

IMAGE FUSION

Feature

Photo Analysis


Detail Extraction Detail Injection

Quality Assessment

Feature Extraction

PanMS

Fused MS

Definition and Classification of Pansharpening

• Pansharpening is the synergic combination of a lower resolution (LR) multispectral

(MS) image and a higher resolution (HR) panchromatic (PAN) image.

• It consists of interpolating the LR MS image to the spatial scale of the HR PAN

image and adding geometrical details extracted from the PAN image.

• Spatial details may be weighted, either globally or locally, to achieve the best

possible enhancement.


possible enhancement.

• Pansharpening methods may be labeled into two main categories, depending on the

way details are extracted from the PAN image:

– methods based on component substitution(CS);

– methods based on multiresolution analysis(MRA).

Pansharpening Based on Component Substitution

• MS+PAN image fusion can be approached as a problem of component substitution:

� apply a reversible transformation to each pixel of MS resampled to PAN scale;

� replace one component with the histogram-matched PAN image;

� inverse transform the resulting data set to yield spatially enhanced MS bands.

• Widely usedspectraltransformationsare:


• Widely usedspectraltransformationsare:

� Intensity-Hue-Saturation (IHS), generalised to more than three bands (GIHS);

� Principal Component Analysis (PCA);

� Gram-Schmidt orthogonalization procedure.

• Advantage of component substitution (CS) based methods isthe outstanding

spatial quality(sharpness and geometric accuracy) of fusion results.

• Drawback isspectral distortion(change in color hues of displayed compositions).

Generalised Component-Substitution Fusion Scheme

+

+w1

w2

-I(Offset)

X

X

g1

G~

B~

G

B

g2


+

+

+w4

w3

Pan

+-

+

I(Offset)

X

X

b

g3

R~

NIR~

NIR^

R

g4

δ

Image Fusion Based on Multiresolution Analysis

• MS+PAN image fusion may be achieved through aredundantimage analysis toavoid artifacts originated fromaliasingand lack of translation invariance:

– Stationary wavelet transform (SWT) with scale ratios being powers of two;

– “A’ trous” wavelet transform (ATWT) generalized to any integer scale ratios

– Laplacian pyramid generalized to fractional scale ratios (GLP)

• A tradeoff between oversampling benefits and processing requirementsis


• A tradeoff between oversampling benefits and processing requirementsisachieved by GLP and by ATWT (Aiazziet al., IEEE T-GRS, Oct. 2002; Ranchinet al. ISPRS J. Photogram. Remote Sensing, June 2003).

• Advantage of multiresolution analysis (MRA) based methods is the spectral

quality (fidelity of colors to original) of fusion results.

• Drawback is thatspatial enhancement(sharpness and geometric accuracy) maybe inadequate.

Multiresolution Analysis (MRA) Fusion Scheme

+

+

+

X

X

g1

G~

B~

G

B

g2


+

+

Pan+

-

+

X

X

g3

R~

NIR~

NIR^

R

g4

δ

−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

10

0.2

0.4

0.6

0.8

1

Normalised along−track frequency

Mag

nitu

de

Normalised across−track frequency

Spectral Weights and Injection Gains of Methods


GS1 patented by Laben and Brower in 2000 and implemented in ENVI.

GIHS proposed by Tu et al. IEEE GRSL Oct. 2004.

GLP-CA proposed by Aiazzi et al.PERS Feb. 2006

GSA proposed by Aiazzi et al.IEEE TGARS Oct. 2007

GSA-CA proposed by Aiazzi et al.IEEE-GRSL Apr. 2009

CS and MRA Fusion

P: PAN imagePL: lowpass filtered PAN imagewk: kth weight of spectral transformation ML(k): kth band of the MS image interpolated at the spatial scale of PAN MF(k): fused kth bandgk: injection gain of kth band


( ) ( ) ( )F L kM k M k P I g= + − ⋅

( ) ( ) ( )F L L kM k M k P P g= + − ⋅1

( )N

k Lk

I w M k=

= ⋅∑

MRA:

CS:

where

Original GeoEye-1 Images (2 m MS, 50 cm PAN)

• May 27 2010: PAN • May 27 2010: MS


Original GeoEye-1 Images (2 m MS, 50 cm PAN)

• July 13 2010: PAN • July 13 2010: MS


Fused Images (1/3)

MRA fusion: MS=July+PAN=July CS fusion: MS=July+PAN=July


Fused Images (2/3)

MRA fusion:MS=July+PAN=May CS fusion: MS=July+PAN=May


Fused Images (3/3)

MRA fusion:MS=May+PAN=July CS fusion: MS=May+PAN=July


Quantitative Analysis

MRA: MS / PAN CS: MS / PAN

Ref=July July/July July/May May/July July/July July/May May/July

ERGAS 2.16 1.90 11.68 2.18 3.71 10.49

SAM (°) 2.19 1.67 11.86 2.18 2.15 11.57

Q4 0.911 0.896 0.323 0.908 0.731 0.453


Ref=May July/July July/May May/July July/July July/May May/July

ERGAS 10.51 10.19 1.85 10.52 9.70 6.15

SAM (°) 12.13 12.00 2.31 12.13 12.11 3.33

Q4 0.406 0.456 0.876 0.403 0.565 0.586

Legend of colors/performances: green-best; red-intermediate; blue poorest

Concluding Remarks

• A unifying framework is introduced to describe the majority of fusion methodsbased oneither component substitution (CS) or multiresolution analysis (MRA).

• Image fusion methods based on component substitution benefit from modeling the spectralresponse of the MS and PAN sensors via a multivariate regression.

• MRA matching the modulation transfer function (MTF) of the MS channels is thekey toachievebestMS+PANimagefusion.


achievebestMS+PANimagefusion.

• CS fusion is very sensitive to temporal misalignment between MS and PAN, i.e., MS andPAN acquired on different times: not only spatial details, but also missing “colors” of thescene portrayed by PAN are partly injected.

• MRA fusion is almost insensitive to temporal misalignment between MS and PAN.

• Conversely, CS fusion is little sensitive to aliasing of the MS image and misregistrationbetween MS and PAN.

Documents

Effects of multitemporal scene changes on pansharpening fusion