J.Y. Tourneret

Post-nonlinear Mixing Models for the Analysis of

Hyperspectral Images

JEAN-YVES TOURNERET

Institut de recherche en informatique de Toulouse (IRIT)

University of Toulouse, France

Joint work with

YOANN ALTMANN AND NICOLAS DOBIGEON

Grenoble 2013 – p. 1/45

J.Y. Tourneret

Outline

Part 1: Linear/ Nonlinear unmixing

Part 2: Polynomial Post-Nonlinear Mixing Model

(PPNMM) for spectral unmixing

Part 3: Nonlinearity detection

Grenoble 2013 – p. 2/45

J.Y. Tourneret

Part 1: Linear/ Nonlinear Unmixing

Hyperspectral Imagery

Grenoble 2013 – p. 3/45

J.Y. Tourneret

Linear Unmixing

Linear mixing model

Reference: Keshava and Mustard, “Spectral unmixing", IEEE Signal Proc.

Magazine, Jan. 2002.Grenoble 2013 – p. 4/45

J.Y. Tourneret

Assumptions for Linear Unmixing

Pure materials sitting side-by-side in the scene

Observation: sum of individual contributions associated

with each component

Single path for the different photons

Valid as a first approximation

Grenoble 2013 – p. 5/45

J.Y. Tourneret

Linear mixing model

y =

R∑

r=1

armr + n = Ma+ n

a = [a1, . . . , aR]T ,M = [m1, ...,mR]

y observed pixel in L bands

R number of pure materials or endmembers (most

spectrally pure vectors)

mr spectrum of the rth endmember

ar abundance of the rth endmember in the pixel

n noise vector n ∼ N (0L, σ2IL)

Grenoble 2013 – p. 6/45

J.Y. Tourneret

Linear mixing model

Physical constraints

Positivity : ar ≥ 0,∀r ∈ 1, ..., R.

Sum-to-one :R∑

r=1

ar = 1.

Grenoble 2013 – p. 7/45

J.Y. Tourneret

Linear mixing model

Unmixing stepsa

Endmember extraction: estimating the spectral

signatures, or endmembers, ( N-FINDRb, VCAc...)

Inversion: estimating the abundances (FCLSd, Bayesian

algo.e...)

aBioucas-Dias et al., “Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based

approaches,” in IEEE JSTARS, April 2012.

bWinter, “An algorithm for fast autonomous spectral end-member determination in hyperspectral data ,” in

Proc. SPIE, 1999.c

Nascimento et al., “Vertex Component Analysis: A fast algorithm to unmix hyperspectral data,” IEEE TGRS,

Apr. 2005.

dHeinz et al., “Fully constrained least-squares linear spectral mixture analysis method for material quantifica-

tion in hyperspectral imagery,” IEEE TGRS, Mar. 2001.e

Dobigeon et al., “Semi-supervised linear spectral unmixing using a hierarchical Bayesian model for hyper-

spectral imagery,” IEEE Trans. Signal Process., Jul. 2008.

Grenoble 2013 – p. 8/45

J.Y. Tourneret

Linear mixing model

Unmixing stepsa

Joint estimation of endmembers and abundances

(DECAb, BLUc...)

aBioucas-Dias et al., “Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based

approaches,” in IEEE JSTARS, April 2012.

bNascimento et al., “Dependent Component Analysis: A Hyperspectral Unmixing Algorithm,” in Proc IGARSS.,

Jul. 2007.c

Dobigeon et al., “Joint Bayesian endmember extraction and linear unmixing for hyperspectral imagery,” IEEE

Trans. Sig. Process., Nov. 2009.

Grenoble 2013 – p. 9/45

J.Y. Tourneret

Nonlinear unmixing

Keshava and Mustard, “Spectral unmixing", IEEE Sig. Proc. Mag., 2002.

Grenoble 2013 – p. 10/45

J.Y. Tourneret

The Generalized Bilinear model

y =R∑

r=1

armr +R−1∑

i=1

R∑

j=i+1

aiajγi,jmi ⊙mj + n

A. Halimi et al, “Nonlinear unmixing of hyperspectral images using a

generalized bilinear model", IEEE Trans. Geo. and Remote Sens., 2011.

Grenoble 2013 – p. 11/45

J.Y. Tourneret

Nonlinear unmixing

Nonlinear mixing models

Possible interactions between the components of the

scene

Nonlinear terms included in the mixing model

Several models depending on the nature of the scene

(intimate mixturesa, bilinear modelsbc,kernel modelsd,...)

aHapke, “Bidirectional reflectance spectroscopy,” J. Geophys. Res., 1981.

bFan and al., “Comparative study between a new nonlinear model and common linear model for analysing

laboratory simulated-forest hyperspectral data,” Remote Sensing of Environment, 2009.c

Halimi and al., “Nonlinear unmixing of hyperspectral images using a generalized bilinear model,” IEEE TGRS,

2011.

dChen and al., “A novel kernel-based nonlinear unmixing scheme of hyperspectral images,” ASILOMAR 2011.,

Nov. 2011.

Grenoble 2013 – p. 12/45

J.Y. Tourneret

Part 2: PPNMM

Post-Nonlinear mixing model

y = g

(R∑

r=1

armr

)+ n = g (Ma) + n

General class of nonlinear models studied for source

separationab

g: unknown linear/nonlinear application

aJutten and Karhunen, “Advances in nonlinear blind source separation,” Proc. 4th Int. Symp. ICA, Apr. 2003.

bBabaie-Zadeh and al., “Separating convolutive post non-linear mixtures,” Proc. 3rd ICA Workshop, Jun.

2001.

Grenoble 2013 – p. 13/45

J.Y. Tourneret

PPNMM

Grenoble 2013 – p. 14/45

J.Y. Tourneret

PPNMM

Mathematical formulation

y = Ma+ b (Ma)⊙ (Ma) + n

n: additive Gaussian noise

b: nonlinearity parameter

M: known matrix of endmembers

a: abundance vector

Remarks

b = 0 : LMM, b 6= 0 : nonlinear model

Weierstrass theorem: gb(s) = s+ bs2

Grenoble 2013 – p. 15/45

J.Y. Tourneret

Parameter Estimation (I)

Bayesian Inferencea

f(a,b, σ2|y) ∝ f(y|a,b, σ2)f(a,b, σ2)

Gaussian likelihood

Appropriate priors to handle the constraints

Estimation algorithm based on MCMC methods

(significant computational complexity)

aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-

spectral imagery,” IEEE Trans. Image Process., to appear.

Grenoble 2013 – p. 16/45

J.Y. Tourneret

Bayesian Estimation (I)

Likelihood

y|a, b, σ2 ∼ N(Ma+ b (Ma)⊙ (Ma) , σ2IL

)

Priors

Abundances

A uniform prior is chosen for a\R = [a1, . . . , aR−1]T

in

the simplex

S =

{a\R

∣∣ar ≥ 0, r ∈ 1, . . . , R − 1 and

R−1∑

r=1

ar ≤ 1

}

withaR = 1−

R−1∑

r=1

ar

Grenoble 2013 – p. 17/45

J.Y. Tourneret

Bayesian Estimation (II)

Priors

Nonlinear coefficient

Conjugate Gaussian distribution

b∣∣σ2

b ∼ N(0, σ2

b

)

Here, σ2b is assigned a conjugate inverse-gamma

distribution σ2b ∼ IG (γ, ν) (where (γ, ν) are fixed real

parameters, i.e., (γ, ν) = (1, 10−2).

Noise variance

Non-informative Jeffreys’ prior

f(σ2) ∝ 1

σ2IR+(σ2)

Grenoble 2013 – p. 18/45

J.Y. Tourneret

Bayesian Model

Joint posteriori distribution of θ ={a\R, b, σ

2, σ2b

}

f(θ|y) ∝ 1

σ2

(1

σ2b

) 32+γ

f(y|a\R, σ2, b) exp

(−b2 + 2ν

2σ2b

)1S(a\R)

Distribution too complex to compute θMMSE ou θMAP.

Simulation Method

A Markov Chain Monte Carlo (MCMC) method is

proposed to sample according to f(θ|y) and to compute

the Bayesian (MMSE or MAP) estimators.

Grenoble 2013 – p. 19/45

J.Y. Tourneret

Principles

Generation of samples θ(1), . . . ,θ(Nr) according to f(θ|y)thanks to a Metropolis-Within-Gibbs sampler

Computation of the MMSE or MAP estimators

θMMSE =1

Nr

Nr∑

i=1

θ(i+Nbi)

θMAP = argθ(i)

max f(θ|y) i = Nbi + 1, . . . , Nbi +Nr

Nbi is the burn-in period and Nr is the number of samples

used for the estimation

Grenoble 2013 – p. 20/45

J.Y. Tourneret

A Metropolis-within-Gibbs Sampler

Four sampling steps

Sampling according to f(a\R|b, σ2, σ2b ,y) using a random

walk (Metropolis-Hastings) truncated in the simplex

Sampling according to f(σ2|a, b, σ2b ,y) using an

inverse-gamma distribution

Sampling according to f(b|a, σ2b , σ

2,y) using a Gaussian

distribution

Sampling according to f(σ2b |a, b, σ2,y) using an

inverse-gamma distribution

Grenoble 2013 – p. 21/45

J.Y. Tourneret

Parameter estimation (II)

Observation model

y = φ (θ) + n

with

φ (θ) = Ma+ b (Ma)⊙ (Ma) and θ = [a, b]T

Maximum likelihood estimator

θ = argminθ

‖y − φ (θ)‖2

subject to the positivity and sum-to-one constraints for a

Non-linear optimization problem with linear constraints!!

Grenoble 2013 – p. 22/45

J.Y. Tourneret

Linearization

Iterative algorithma

Taylor Expansion (linearization) of φ at θ(t)

φ(θ) = φ(θ(t))+∇φ

(θ(t))(

θ − θ(t))+ ǫ

Recursive minimization

θ(t+1) = argminθ

∥∥∥y − φ(θ(t))−∇φ

(θ(t))(

θ − θ(t))∥∥∥

2

subject to the positivity and sum-to-one constraints for a

Convergence difficult to prove

aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-

spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.

Grenoble 2013 – p. 23/45

J.Y. Tourneret

Steepest descent

Principlesa

Reparametrization

φ (θR) = Ma+ b (Ma)⊙ (Ma) with θR = [a−R, b]T

Steepest descent algorithm

θ(t+1)R = θ

(t)R − λ∇J

(θ(t)R

)and J (θR) =

1

2‖y − φ (θR)‖2

where λ is chosen to satisfy the constraints (line search)

Convergence toward a local minimum

aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-

spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.

Grenoble 2013 – p. 24/45

J.Y. Tourneret

A Toy Example

Pure materials: Galvanized Steel Metal (red), Green Grass(blue) and Olive Green Paint (green)

Grenoble 2013 – p. 25/45

J.Y. Tourneret

A Toy Example

Parameters: (a1, a2, a3) = (0.3, 0.6, 0.1), b = 0.3, SNR =15dB.

Grenoble 2013 – p. 26/45

J.Y. Tourneret

MMSE Estimators with Confidence Intervals

Grenoble 2013 – p. 27/45

J.Y. Tourneret

PPNMM Validation

I1 I2 I3 I4

(LMM) (FM) (GBM) (PPNMM)

FCLS (LMM) 1.58 24.72 9.49 16.87

Taylor (FM) 22.67 1.49 12.61 26.33

Taylor (GBM) 6.32 14.67 7.07 15.61

Taylor (PPNMM) 2.70 3.83 3.26 3.33

Abundance RMSEs (×10−2): synthetic imagesa

aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-

spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.

Grenoble 2013 – p. 28/45

J.Y. Tourneret

PPNMM Validation

I1 I2 I3 I4

(LMM) (FM) (GBM) (PPNMM)

FCLS (LMM) 5.28 5.74 5.42 5.48

Taylor (FM) 5.61 5.28 5.38 5.75

Taylor (GBM) 5.31 5.40 5.30 5.42

Taylor (PPNMM) 5.29 5.29 5.28 5.28

Reconstruction errors (×10−2): synthetic imagesa

aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-

spectral imagery,” IEEE Trans. Image Process., vol. 21, no. 6, pp. 3017-3025, June 2012.

Grenoble 2013 – p. 29/45

J.Y. Tourneret

Estimated Nonlinearity Parameter

-0.2 -0.1 0 0.1 0.20

5

10

15

20

I1

b-0.2 0 0.2 0.40

2

4

6

8

I2

b

-0.2 0 0.2 0.40

2

4

6

I3

b-0.4 -0.2 0 0.2 0.40

0.5

1

1.5

2

I4

b

Grenoble 2013 – p. 30/45

J.Y. Tourneret

Real data: Cuprite Image

AVIRIS Cuprite image of 190× 250 pixels (composite natural colors).

R = 14 endmembers, L = 189 spectral bands.Grenoble 2013 – p. 31/45

J.Y. Tourneret

Nonlinearity Parameter (Cuprite)

Grenoble 2013 – p. 32/45

J.Y. Tourneret

Reconstruction Errors (Cuprite)

Table 1: Reconstruction error for the Cuprite image.

RE (×10−2)

LMM FM GBM PPNMM

2.11 3.03 2.02 1.19

Grenoble 2013 – p. 33/45

J.Y. Tourneret

Part 3: Nonlinearity detection

Binary hypothesis testinga

H0 : y is distributed according to the LMM (b = 0)

H1 : y is distributed according to the PPNMM (b 6= 0)

Statistical properties of the nonlinearity parameter estimator

H0 : b ∼ N (0, s20)

H1 : b ∼ N (b, s21)

where s20 is a function of (a, σ2) and s21 depends on (a, b, σ2).

aAltmann and al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model for hyper-

spectral imagery,” IEEE Trans. Image Process., vol. 22, no. 4, pp. 1267-1276, April 2013.

Grenoble 2013 – p. 34/45

J.Y. Tourneret

Known parameters a and σ2

Generalized Likelihood Ratio Test (GLRT)

Definition

supb

p(b|H1)

p(b|H0)

H1

≷H0

ν ⇔ T 2 =b2

s20

H1

≷H0

η

where η is an appropriate threshold related to the PFA.

Probability of false alarm and probability of detection

PFA = 2φ(−√η)

PD(b) = 1 + φ

(−s0√η − b

s1

)− φ

(s0√η − b

s1

)

where φ(·) is the cdf of the N (0, 1) Gaussian distribution.

Grenoble 2013 – p. 35/45

J.Y. Tourneret

Unknown parameters a and σ2

Maximum Likelihood estimator of a, b and σ2

θ = (a, b, σ2)

Detection strategy

T 2 =b2

s20

H1

≷H0

η with s20 = CCRLB(b = 0; a, σ2)

Threshold determination

T ∼ N (0, 1)

Grenoble 2013 – p. 36/45

J.Y. Tourneret

Cramér-Rao Lower bound (CRLB)

Fisher information matrix of θ = [aT , b, σ2]T

[JF ]i,j = −E

[∂2 ln f(y|θ)

∂θi∂θj

]i, j = 1, . . . , R + 2

Unconstrained CRLB B

βi,j = [B]i,j =[J−1

F

]i,j

i, j = 1, ..., R + 2

Constrained CRLBa of any unbiased estimator of b under H0

CCRLB(a, σ2) = βR+1,R+1 −(

R∑

i=1

R∑

j=1

βi,j

)(R∑

j=1

βR+1,j

)2

aGorman and Hero, “Lower bounds for parametric estimation with constraints,” IEEE Trans. IT, Nov. 1990.

Grenoble 2013 – p. 37/45

J.Y. Tourneret

Unknown parameters a and σ2

Left: Variance of b under H0 (blue crosses) and CRLB (black line) versus

σ2.Right: Distribution of T under H0 for σ2 = 10−3 (black line) and

standard Gaussian probability density function (red line). R = 3

endmembers, L = 826 spectral bands, a = [0.3, 0.6, 0.1]T .

Grenoble 2013 – p. 38/45

J.Y. Tourneret

Simulation results

Synthetic data (50× 50 pixels)

Four images (4 mixing models)

S1 : LMM

S2 : Bilinear Fan model (γi,j = 1)

y =

R∑

r=1

armr +

R−1∑

i=1

R∑

j=i+1

γi,jaiajmi ⊙mj + n

S3 : Generalized Bilinear model (γi,j ∼ U(0;1))

S4 : PPNMM (b ∼ U(−0.3;0.3))

R = 3 endmembers, L = 826 spectral bands, a uniform in the simplex.

Grenoble 2013 – p. 39/45

J.Y. Tourneret

Synthetic Data

Left: the four sub-images S1 (LMM), S2 (FM), S3 (GBM) and S4 (PPNMM).

Right: Detection maps using PFA = 0.05. Black (resp. white) pixels

correspond to pixels detected as linearly (resp. nonlinearly) mixed.

Grenoble 2013 – p. 40/45

J.Y. Tourneret

Synthetic data

Pixels detected as linear (red crosses) and nonlinear (blue dots) for the

four sub-images S1 (LMM), S2 (FM), S3 (GBM) and S4 (PPNMM).

Grenoble 2013 – p. 41/45

J.Y. Tourneret

Real data: Cuprite Image

AVIRIS image of 190× 250 pixels extracted from Cuprite image observed

in composite natural colors.

R = 14 endmembers, L = 189 spectral bands.

Grenoble 2013 – p. 42/45

J.Y. Tourneret

Real data: Cuprite Image

Left: map of b for the Cuprite image. Detection maps for PFA = 10−3

(middle) and for PFA = 10−4 (right). Black (resp. white) pixels correspond

to pixels detected as linearly (resp. nonlinearly) mixed.

Grenoble 2013 – p. 43/45

J.Y. Tourneret

Conclusions

Post nonlinear models have shown promising results for

spectral unmixing in the presence of nonlinear mixtures

pixel-by-pixel nonlinearity detection to determine which

parts of an hyperspectral image are characterized by

nonlinear mixtures.

Future work within the ANR HYPANEMA

Derive a fully unsupervised algorithm for abundance and

endmember estimation for nonlinear mixtures

Introduction of spatial correlation between adjacent pixels

for parameter estimation and nonlinearity detection

Grenoble 2013 – p. 44/45

J.Y. Tourneret

An Example (Ultrasound Images)

(a) (b) (c)

Actual location of skin lesion (left) and the corresponding estimated labels

(healthy = white, lesion = red) without spatial correlation (middle) and with

spatial correlation (right)a.

aPereyra et al., “Segmentation of skin lesions in 2D and 3D ultrasound images using a spatially coherent

generalized Rayleigh mixture model,” IEEE Trans Med Imaging., Mar. 2012.

Grenoble 2013 – p. 45/45