Robustness of Multiway Methods in Relation to Homoscedastic and Hetroscedastic Noise

1

Robustness of Multiway Methods Robustness of Multiway Methods in Relation to Homoscedastic in Relation to Homoscedastic

and Hetroscedastic Noiseand Hetroscedastic Noise

T. Khayamian T. Khayamian

Department of Chemistry , Isfahan University of Department of Chemistry , Isfahan University of

Technology, Isfahan 84154, IranTechnology, Isfahan 84154, Iran

2

OutlineOutline IntroductionIntroduction Prediction of component concentrations in Claus Prediction of component concentrations in Claus

data using PARAFAC and N-PLS multi-way methods data using PARAFAC and N-PLS multi-way methods original dataoriginal data (original + noise) data (original + noise) data denoised data (using wavelet as a denoising method)denoised data (using wavelet as a denoising method) - Homoscedastic noise (level independent method)- Homoscedastic noise (level independent method) - Hetroscedastic noise (level dependent and minimum - Hetroscedastic noise (level dependent and minimum

description length)description length)

ConclusionsConclusions

3

Noise definitionNoise definition

● Noise is any component of a signal Noise is any component of a signal which impedes observation, detection which impedes observation, detection or utilization of the information that or utilization of the information that the signal is carrying. the signal is carrying.

● Noise is measured by its standard Noise is measured by its standard deviation or peak to peak fluctuationdeviation or peak to peak fluctuation

4

Different types of noiseDifferent types of noise

HetroscedasticHetroscedastic

HomoscedasticHomoscedastic

NoiseNoise

5

Homoscedastic and Homoscedastic and Hetroscedastic NoiseHetroscedastic Noise

● Homoscedastic noise: Homoscedastic noise: Noise is independent of variable, sample and Noise is independent of variable, sample and

signal with the normal distribution and a signal with the normal distribution and a constant variance. constant variance.

● Hetroscedastic Noise:Hetroscedastic Noise: Noise is dependent on the variable, sample Noise is dependent on the variable, sample

and signal.and signal. (Noise from different variables or samples (Noise from different variables or samples

can be correlated) can be correlated)

11 22

RR 11

22

22

22

6

Least squares methodLeast squares method

m

i

n

jcalijij yyS

1 1

2)((exp)

2

Homoscedastic noise :

ij is constant, uniform and independent of the signal, variables and samples

m

i

n

j ij

calijij yyS

1 1

2

)((exp)2

Hetroscedastic noise :

ij is dependent on signal, variables or samples

m

i

n

j ij

calijij yyS

1 1

2

)((exp)2

7

Hetroscedastic noise in Univariate Hetroscedastic noise in Univariate and Multivariate Calibration and Multivariate Calibration

MethodsMethods● Zeroth order calibrationZeroth order calibration weighted linear regressionweighted linear regression● First order calibrationFirst order calibration weighted principle component weighted principle component

analysisanalysis ● Second order calibrationSecond order calibration Positive matrix factorization Positive matrix factorization Maximum likelihood PARAFACMaximum likelihood PARAFAC

2'* TPXW

2'* TPXW

8

Claus data “fluorescence Claus data “fluorescence Spectroscopy” Spectroscopy”

Analyte Analyte 11(tyrosine)(tyrosine)

Analyte 2Analyte 2(tyrptophan(tyrptophane)e)

Analyte 3Analyte 3(phenyl (phenyl alanine)alanine)

Sample Sample 11

2.7×12.7×100-6-6

00 00

Sample Sample 22

00 1.33×101.33×10--

55

00

Sample Sample 33

00 00 9.0×109.0×10-4-4

Sample Sample 44

1.6×11.6×100-6-6

5.4×105.4×10-6-6 3.55×103.55×10-4-4

Sample Sample 55

9.0×19.0×100-7-7

4.4×104.4×10-6-6 2.97×102.97×10-4-4

C. A. Andesson and R. Bro. The N-way Toolbox for MATABC. A. Andesson and R. Bro. The N-way Toolbox for MATABChemom. Intell. Lab. Sys. 2000, 52 (1), 1- 4Chemom. Intell. Lab. Sys. 2000, 52 (1), 1- 4http://www. models . kvl. dkhttp://www. models . kvl. dk

9

Fluorescence excitation and emission spectrum of five samplesFluorescence excitation and emission spectrum of five samples

10

XX44

201201

6161

==

++++aa11

bb11

cc11 cc22 cc33

aa22

bb22 bb33

aa33

Claus dataClaus data

PARAFAC: four samples were used for modeling PARAFAC: four samples were used for modeling

••••••••

••

ScoreScore (a1)(a1)

Concentration analyte 1Concentration analyte 1

••••••••

••

ScoreScore (a2)(a2)


••••••••

••

ScoreScore (a3)(a3)


11

Calculation of Score for a New Calculation of Score for a New SampleSample

6161

201201

Z = kr (B, C)Z = kr (B, C)Un = reshape (Un, 12261, 1)Un = reshape (Un, 12261, 1)Score Un = pinv(Z) * UnScore Un = pinv(Z) * Un

Un =Un =

12

Relative Errors of Predicted Relative Errors of Predicted Concentrations for Samples Concentrations for Samples 4 & 5 4 & 5

(without adding noise)(without adding noise)

Analyte 1Analyte 1 Analyte 2Analyte 2 Analyte 3Analyte 3

Sample 4Sample 4 -0.75-0.75 -7.3-7.3 9.49.4

Sample 5Sample 5 -0.56-0.56 -6.1-6.1 14.314.3

13

Generating of Noise Matrix

55

201201×61×61

(Claus data)(Claus data) 55 NoiseNoise

Homoscedastic nois:Homoscedastic nois:Standard deviation of noise = 2%, 5%, 10% of the maximum value in the claus dataStandard deviation of noise = 2%, 5%, 10% of the maximum value in the claus data

Hetroscedastic noise :Hetroscedastic noise :

N = N(0,1) .N = N(0,1) .* * 1/10 X 1/10 X

Element by element was multiplied by one-tenth of the claus data Element by element was multiplied by one-tenth of the claus data

201201×61×61

Claus data + NoiseClaus data + Noise

44

201201

6161unfoldingunfolding

14

0 2000 4000 6000 8000 10000 12000 14000-200

-150

-100

-50

0

50

100

150homoscedastic noise (10%) for sample one

201 X 610 2000 4000 6000 8000 10000 12000 14000

-80

-60

-40

-20

0

20

40

60

80hetroscedastic noise (10%) for sample one

201 X 61

Homoscedastic and Hetroscedastic noise were added to original dataHomoscedastic and Hetroscedastic noise were added to original data

Hetroscedastic noise (10%)Hetroscedastic noise (10%)Homoscedastic noise (10%)Homoscedastic noise (10%)

15

0 2000 4000 6000 8000 10000 12000 14000-200

0

200

400

600

800

1000signal + noise (homo) for sample one

201 X 610 2000 4000 6000 8000 10000 12000 14000

-100

0

100

200

300

400

500

600

700

800reshape of sample one

201 X 61

Reshape of Sample One Reshape of Sample One Sample one with adding Homoscedastic Sample one with adding Homoscedastic noisenoise

The effect of adding Homoscedastic noiseThe effect of adding Homoscedastic noise

16

0 2000 4000 6000 8000 10000 12000 14000-100

0

100

200

300

400

500

600

700

800reshape of sample one

201 X 610 2000 4000 6000 8000 10000 12000 14000

-100

0

100

200

300

400

500

600

700

800signal + noise(het,20%) for sample one

201 X 61

Reshape of Sample One Reshape of Sample One Sample one with adding Hetroscedastic Sample one with adding Hetroscedastic noisenoise

The effect of adding Hetroscedastic noiseThe effect of adding Hetroscedastic noise

17

Wavelet can be used as a Wavelet can be used as a powerful tool for signal powerful tool for signal

denoising denoising Wavelet Denoising :Wavelet Denoising :● Wavelet decomposition of the signalWavelet decomposition of the signal● Selecting the threshold Selecting the threshold ● Applying the threshold to the wavelet Applying the threshold to the wavelet

coefficientscoefficients● Inverse transformation to the native Inverse transformation to the native

domaindomain

18

Thresholding methods :Thresholding methods :

● Global thresholdingGlobal thresholding● Level dependent thresholdingLevel dependent thresholding● Data dependent thresholdingData dependent thresholding● Cycle – spin thresholdingCycle – spin thresholding● Wavelet packet thresholdingWavelet packet thresholding

19

Universal threshold :Universal threshold :

N = length of data array

6745.0

)( ixmedian

Xi = detail part of coefficient

)ln(2 Nt

20

0 2000 4000 6000 8000 10000 12000 14000 16000 18000-100

0

100

200

300

400

500

600

700

800mirror padding for sample one

16384=214

21

Prediction of Analyte Concentrations for Prediction of Analyte Concentrations for

Samples 4 & 5 Samples 4 & 5 using using PARAFACPARAFAC

22

Comparison of Sum of the Square of Comparison of Sum of the Square of Residuals (Homoscedastic noise - Residuals (Homoscedastic noise -

PARAFAC)PARAFAC)SSRSSR

Model 1Model 1 SSRSSR

Model 2Model 2

Without noiseWithout noise 10.510.5 10.210.2

Noisy Noisy datadata

Homo. noise Homo. noise 2%2%

168.56168.56 168.57168.57


168.56168.56 168.57168.57


168.56168.56 168.57168.57

Denoised Denoised datadata


14.3414.34 15.0515.05


37.2437.24 37.2937.29


105.36105.36 103.97103.97

model 1 : sample 1, 2 , 3, 4 / model 2 : sample 1, 2, 3, 5model 1 : sample 1, 2 , 3, 4 / model 2 : sample 1, 2, 3, 5 Each number × 10Each number × 1055

23

Var. Var.

Model 1Model 1 Var.Var.

Model 2Model 2Without noiseWithout noise 99.9499.94 99.9599.95


Homo. noise Homo. noise 2%2% 99.0799.07 99.1999.19





99.9399.93 99.9299.92


99.8299.82 99.7999.79


99.4999.49 99.4299.42

Comparison of explained variation Comparison of explained variation (Homoscedastic noise - PARAFAC)(Homoscedastic noise - PARAFAC)

24

Relative Errors of Predicted Concentrations Relative Errors of Predicted Concentrations for Sample 4 for Sample 4

( Homoscedastic noise – PARAFAC ) ( Homoscedastic noise – PARAFAC )

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-0.75-0.75 -7.3-7.3 9.49.4


2 % 2 % noisenoise

-0.41-0.41 -7.1-7.1 9.29.2

5 % 5 % noise noise

-0.41-0.41 -7.1-7.1 9.269.26

10 % 10 % noisenoise

-0.41-0.41 -7.1-7.1 9.269.26

DenoiseDenoised datad data

2 % 2 % noisenoise

-0.77-0.77 -7.5-7.5 9.39.3

5 % 5 % noise noise

-0.66-0.66 -7.5-7.5 8.58.5


-1.3-1.3 -7.1-7.1 7.37.3

25

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-0.75-0.75 -7.3-7.3 9.49.4


2 % 2 % noisenoise

-0.71-0.71 -6.1-6.1 14.214.2

5 % 5 % noise noise

-0.71-0.71 -6.1-6.1 14.214.2


-0.71-0.71 -6.1-6.1 16.916.9


2 % 2 % noisenoise

-0.55-0.55 -6.1-6.1 14.314.3

5 % 5 % noise noise

-1.7-1.7 -6.4-6.4 13.613.6


-2.8-2.8 -5.3-5.3 11.611.6


( Homoscedastic noise - PARAFAC ) ( Homoscedastic noise - PARAFAC )

26

Comparison of Sum of the Square of Comparison of Sum of the Square of Residuals (Hetroscedastic noise)Residuals (Hetroscedastic noise)

SSRSSR

Model Model 11

SSRSSR

Model 2Model 2

Without noiseWithout noise 10.510.5 10.210.2Noisy Noisy datadata

Hetro. noise Hetro. noise 10%10%

35.5235.52 39.2639.26


110.27110.27 126.17126.17



33.5233.52 36.5636.56


101.32101.32 113.78113.78(Each number * 10(Each number * 1055)) wavelet denoising (level dependent method)wavelet denoising (level dependent method)

27

Comparison of explained variation Comparison of explained variation (Hetroscedastic noise - PARAFAC)(Hetroscedastic noise - PARAFAC)

Var.Var.

Model Model 11

Var.Var.

Model 2Model 2



99.8099.80 99.8199.81


99.3999.39 99.3999.39



99.8199.81 99.8299.82


99.4499.44 99.4599.45wavelet denoising (level dependent method)wavelet denoising (level dependent method)

28

Relative Errors of Predicted Concentrations Relative Errors of Predicted Concentrations for sample 4 for sample 4

( Hetroscedastic noise - PARAFAC) ( Hetroscedastic noise - PARAFAC)

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-0.7-0.7 -7.3-7.3 9.49.4



-0.85-0.85 -7.3-7.3 9.39.3


-0.94-0.94 -7.0-7.0 9.39.3



-0.86-0.86 -7.16-7.16 9.289.28


-0.94-0.94 -7.10-7.10 9.159.15

wavelet denoising (level dependent method)wavelet denoising (level dependent method)

29



Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-0.56-0.56 -6.1-6.1 14.314.3



-0.53-0.53 -6.1-6.1 14.214.2


-0.45-0.45 -6.2-6.2 14.114.1



-0.54-0.54 -6.2-6.2 14.1514.15


-0.47-0.47 -6.04-6.04 14.0014.00

wavelet denoising (level dependent method)wavelet denoising (level dependent method)

30

Minimum Description LengthMinimum Description Length

• The MDL is an approach to simultaneous noise suppression The MDL is an approach to simultaneous noise suppression and signal compression.and signal compression.

• It is free from any parameter setting such as threshold It is free from any parameter setting such as threshold selection, which can be particularly useful for real data selection, which can be particularly useful for real data where the noise level is difficult to estimate. where the noise level is difficult to estimate.

m = filter typem = filter typellmm = the number of major coefficients retained = the number of major coefficients retained

γγjj,k,k = the vector of wavelet coefficients of transformed type m = the vector of wavelet coefficients of transformed type m

γγj,kj,k = = the vector of the contractedthe vector of the contracted wavelet coefficientswavelet coefficients

2

22

3 mlkj

mkj

NNlmlMDL ,,loglogmin),(

mlml

31

Signal Denoising with MDL Signal Denoising with MDL methodmethod

0 5000 10000 15000-200

0

200

400

600

800

1000Raw Data

0 0.5 1 1.5 2

x 104

-200

0

200

400

600

800

1000Noisy Data

0 0.5 1 1.5 2

x 104

2

4

6

8

10

12

14x 10

4 MDL

0 0.5 1 1.5 2

x 104

-200

0

200

400

600

800

1000Denoised Data

32

Comparison of Sum of the Square of Comparison of Sum of the Square of Residuals (Hetroscedastic noise - Residuals (Hetroscedastic noise -


Model Model 11

SSRSSR

Model 2Model 2



35.5235.52 39.2639.26


110.27110.27 126.17126.17



35.5235.52 39.2539.25


110.26110.26 126.17126.17Each number × 10Each number × 1055 Wavelet Denoising (MDL)Wavelet Denoising (MDL)

33

Comparison of explained variation Comparison of explained variation (Hetroscedastic noise - PARAFAC)(Hetroscedastic noise - PARAFAC)

Var.Var.

Model Model 11

Var.Var.

Model 2Model 2



99.8099.80 99.8199.81


99.3999.39 99.3999.39



99.8099.80 99.8199.81


99.3999.39 99.3999.39Wavelet Denoising (MDL)Wavelet Denoising (MDL)

34

Relative Errors of Predicted Relative Errors of Predicted Concentrations for sample 4 Concentrations for sample 4


Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-0.75-0.75 -7.3-7.3 9.49.4



-0.85-0.85 -7.3-7.3 9.39.3


-0.94-0.94 -7.0-7.0 9.39.3



-0.85-0.85 -7.28-7.28 9.39.3


-0.94-0.94 -7.01-7.01 9.39.3

Wavelet Denoising (MDL)Wavelet Denoising (MDL)

35

Relative Errors of Predicted Relative Errors of Predicted Concentrations for Sample 5 Concentrations for Sample 5

( Hetroscedastic noise - PARAFAC ) ( Hetroscedastic noise - PARAFAC )

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-0.5-0.5 -6.1-6.1 14.314.3



-0.5-0.5 -6.1-6.1 14.214.2


-0.4-0.4 -6.2-6.2 14.114.1



-0.5-0.5 -6.1-6.1 14.214.2


-0.4-0.4 -6.2-6.2 14.114.1


36

Prediction of Analyte Concentrations for Prediction of Analyte Concentrations for

Samples 4 & 5 Samples 4 & 5 using using N-PLSN-PLS

37


( Homoscedastic noise – NPLS model)( Homoscedastic noise – NPLS model)

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-2.88-2.88 -0.44-0.44 4.874.87


2 % 2 % noisenoise

-2.91-2.91 -0.39-0.39 4.914.91

5 % 5 % noise noise

-2.97-2.97 -0.33-0.33 4.954.95


-3.07-3.07 -0.22-0.22 4.994.99


2 % 2 % noisenoise

-2.91-2.91 -0.44-0.44 4.924.92

5 % 5 % noise noise

-3.00-3.00 -0.20-0.20 4.674.67


-3.29-3.29 0.360.36 4.364.36

X-block > 99X-block > 99 Y-block > 99Y-block > 99

38


( Homoscedastic noise – NPLS model ) ( Homoscedastic noise – NPLS model )

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-1.67-1.67 0.440.44 11.2411.24


2 % 2 % noisenoise

-1.65-1.65 0.350.35 11.3811.38

5 % 5 % noise noise

-1.62-1.62 0.220.22 11.5711.57


-1.57-1.57 -0.01-0.01 11.8611.86


2 % 2 % noisenoise

-1.69-1.69 0.310.31 11.3411.34

5 % 5 % noise noise

-2.08-2.08 0.270.27 11.5611.56


-2.89-2.89 0.930.93 10.6610.66

X-block > 99X-block > 99 Y-block > 99Y-block > 99

39

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-2.88-2.88 -0.44-0.44 4.874.87



-3.00-3.00 -0.28-0.28 4.894.89


-3.12-3.12 -0.12-0.12 4.914.91



-3.00-3.00 -0.28-0.28 4.914.91


-3.12-3.12 -0.12-0.12 4.914.91


( Hetroscedastic noise – NPLS model) ( Hetroscedastic noise – NPLS model)

Wavelet Denoising (MDL)Wavelet Denoising (MDL)X-block > 99X-block > 99 Y-block > 99Y-block > 99

40

Comparison of Sum of the Square of Comparison of Sum of the Square of Residuals (Hetroscedastic noise - Residuals (Hetroscedastic noise -


Model Model 11

SSRSSR

Model 2Model 2



35.5235.52 39.2639.26


110.27110.27 126.17126.17



33.5233.52 36.5636.56


101.32101.32 113.78113.78Each number × 10Each number × 1055

41

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-2.88-2.88 -0.44-0.44 4.874.87



-3.00-3.00 -0.28-0.28 4.894.89


-3.12-3.12 -0.12-0.12 4.914.91



-3.00-3.00 -0.36-0.36 4.884.88


-3.12-3.12 -0.23-0.23 5.785.78


( Hetroscedastic noise – NPLS model) ( Hetroscedastic noise – NPLS model)

42

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-1.67-1.67 0.440.44 11.2411.24



-1.61-1.61 0.430.43 11.2311.23


-1.56-1.56 0.400.40 11.2111.21



-1.61-1.61 0.420.42 11.1911.19


-1.56-1.56 0.390.39 11.1511.15


( Hetroscedastic noise - NPLS model )( Hetroscedastic noise - NPLS model )

43

Analyte Analyte 11

Analyte Analyte 22

Analyte Analyte 33

0 % 0 % noisenoise

-1.67-1.67 0.440.44 11.2411.24



-1.61-1.61 0.430.43 11.2311.23


-1.56-1.56 0.400.40 11.2111.21



-1.61-1.61 0.430.43 11.2111.21


-1.56-1.56 0.400.40 11.2111.21


( Hetroscedastic noise – NPLS model )( Hetroscedastic noise – NPLS model )


44

X-Block X-Block

Model Model 11

Y-Block Y-Block

Model 1Model 1

X-Block X-Block

Model Model 22

Y-Block Y-Block

Model 2Model 2

noise noise 0%0% 99.9499.94 99.8599.85 99.9399.93 99.9499.94


noise noise 2%2% 99.8099.80 99.8599.85 99.7999.79 99.9499.94

noise noise 5%5% 99.0699.06 99.8599.85 99.0499.04 99.9499.94

noise noise 10%10% 96.5096.50 99.8499.84 96.5196.51 99.9499.94

DenoisDenoised dataed data

noise noise 2%2%

99.8599.85 99.9799.97 99.9199.91 99.8599.85

noise noise 5%5%

99.6799.67 99.9799.97 99.7799.77 99.8599.85

noise noise 10%10%

99.1099.10 99.9799.97 99.3299.32 99.8899.88

Comparison of explained variation Comparison of explained variation (Homoscedastic noise – NPLS model)(Homoscedastic noise – NPLS model)

45

Comparison of explained variation Comparison of explained variation (Hetroscedastic noise – NPLS model)(Hetroscedastic noise – NPLS model)

X-Block X-Block

Model Model 11

Y-Block Y-Block

Model 1Model 1

X-Block X-Block

Model Model 22

Y-Block Y-Block

Model 2Model 2

noise noise 0%0% 99.9499.94 99.8599.85 99.9399.93 99.9499.94


noise noise 10%10% 99.8099.80 99.8599.85 99.7899.78 99.9499.94

noise noise 20%20% 99.3899.38 99.8599.85 99.3499.34 99.9499.94


noise noise 10%10%

99.8199.81 99.8599.85 99.7999.79 99.9499.94

noise noise 20%20%

99.4399.43 99.8699.86 99.3999.39 99.9499.94

46

Comparison of explained variationComparison of explained variation (Hetroscedastic noise – NPLS model) (Hetroscedastic noise – NPLS model)

X-Block X-Block

Model Model 11

Y-Block Y-Block

Model 1Model 1

X-Block X-Block

Model Model 22

Y-Block Y-Block

Model 2Model 2

noise noise 0%0% 99.9499.94 99.8599.85 99.9399.93 99.9499.94


noise noise 10%10% 99.8099.80 99.8599.85 99.7899.78 99.9499.94

noise noise 20%20% 99.3899.38 99.8599.85 99.3499.34 99.9499.94


noise noise 10%10%

99.8099.80 99.8599.85 99.7899.78 99.9499.94

noise noise 20%20%

99.3899.38 99.8599.85 99.3499.34 99.9499.94


Documents

Robustness of Multiway Methods in Relation to Homoscedastic and Hetroscedastic Noise