Correction of spurious trends in climate series caused by inhomogeneities Ralf Lindau

Correction of spurious trends

in climate series caused by inhomogeneities

Ralf Lindau

Dipdoc Seminar – 12. November 2015

Break detectionConsider the differences of one station

compared to a neighbor reference from a surrounding network of stations.

The dominating natural variance is cancelled out, because it is very similar at both stations.

Breaks become visible by abrupt changes in the station-reference time series.

Internal variance (Noise)within the subperiods

External variance (Signal)between the means of different

subperiods

Break criterion:Maximum external (explained) variance

Break-aware ideaBreaks are only detected, but not corrected.

Calculate the mean trend over all homogeneous subperiods (omitting the known breakpoints).

This trend should reflect the true trend.


𝑡 𝑟𝑒𝑛𝑑=𝑁 2−1𝑁 2 𝑡𝑒𝑥𝑡+

1𝑁2 𝑡 𝑖𝑛𝑡Is not promising, because:

Correction is important

Known break positions alone are not sufficient to derive reliable trends.

The correction part of homogenization determines a fraction of of the trend, when N is the number of subsegments.

Even for only one break this is equal to ¾.


Station or network trend?

From my point of view, the network-mean (regionally averaged) trend is per se more interesting.

Moreover, trend corrections for individual stations are easy to derive, if the network-mean trend is known.

Therefore, we concentrate on corrections of the network-mean trend.


Network-mean trend error


𝑇 𝑖0=𝐶+𝐼𝑖 0Observed = True + Spurious station trend

1𝑛∑𝑖=1

𝑛

(𝑇 𝑖 0−𝑇 𝑖 )=1𝑛∑𝑖=1

𝑛

(𝐶+𝐼 𝑖0−𝐶− 𝐼 𝑖 )=𝐼 𝑖0−1𝑛∑𝑖=1

𝑛

𝐼 𝑖

Observed trend anomaly = Spurious station trend against the network-mean – network-mean trend errortrend

Two ways of correction

1. Break-by-break method:Consider an individual break of one candidate station.

Compare the two homogeneous subperiods before and after this break with homogeneous subperiods of suited

neighbor stations, which preferably long overlap periods with the candidate break.

2. ANOVA method:Minimize the variance of the entire network.Discussed in the following, because it is better defined.

Dipdoc Seminar – 16. June 2014

ANOVA correction scheme (1/3)

Observation b (station, year)Climate signal c (year)Inhomogeneity a (station, year)Noise e (station, year)

Minimize the squared difference between theory and observations.

Derivation with respect to c(j) leads to m equations, one for each c(j).


𝑏 (𝑖 , 𝑗 )=𝑐 ( 𝑗 )+𝑎 (𝑖 , 𝑗 )+𝜀 (𝑖 , 𝑗 )

∑𝑖=1

𝑛

∑𝑗=1

𝑚

(𝑐 ( 𝑗 )+𝑎 (𝑖 , 𝑗 )−𝑏 (𝑖 , 𝑗 ) )2=min

𝑐 ( 𝑗 )= 1𝑛∑𝑖=1𝑛

(𝑏 (𝑖 , 𝑗 )−𝑎 (𝑖 , 𝑗 ) )=0


Analogously, we get h equations, one for each of the homogeneous sub-periods.

Altogether we have m+h equations for m+h unknowns.

Thus a (m+h) x (m+h) matrix has to be solved.E.g. 115x115-matrix for 100 years and 15 subperiods.

We can insert the m climate equations for each year into the h equations for each subperiod, so that only a hxh-matrix has to solved.



n = 5 stations with data from 100 years.h = 15 homogeneous sub-periods in total.

Consider sub-period no. 8:

It has an overlap of 32 years with a2,of 13 years with a4,of 19 years with a5, etc.

The overlap periods constitutes the non-diagonal matrix elements.

The diagonal is given by (n-1) length(ai)


Simulated data

Test the performance of the ANOVA correction scheme with simulated data.

The simulated data consist of three superimposed signals:

1. The climate signal, which identical for all stations of a network.

2. Noise, which mimics the difference between the stations, e.g. due to weather.

3. Inhomogeneities inserted at random timings and with random strengths.


Test under perfect conditions

1000 networks10 stations with 5 breaks each

No mean trend errorNo noiseKnown break positions

Result:

Perfect skill.- We made no programming errors.- The method works perfectly.


Inserted yearly station inhomogeneity

Det

ecte

d ye

arly

sta

tion

inho

mog

enei

ty

Signal / noise = 1

Still: No mean trend errorPerfectly known break positions

Equal break and noise variance:SNR = 1

Result:

As expected no longer perfect, but r = 0.955


Inserted yearly station inhomogeneity

Det

ecte

d ye

arly

sta

tion

inho

mog

enei

ty

Network-mean trends

From individual inhomogeneities to network-mean trends.Both regressions are calculated.That taking the x-axis data as independent is in all three cases equal to the 1-to-1 line.

What does this mean?


Inhomogeneities Station trends Network trends

r = 0.9550 r = 0.9525 r = 0.8092

What does it mean?

It means that:

y is equal to x plus random scatter e.

Because in this case, the following equation chain holds true:


Remaining trend error

It is convenient to display not the detected (y), but the remaining (y-x) trend error.

As shown the inserted and the remaining quantities are uncorrelated.

The remaining errors are smaller, but comparable in size.

This is valid for SNR = 1


Inserted network-mean trend error

Rem

aini

ng n

etw

ork-

mea

n tr

end

erro

r

sx2 = 0.265

sy2 = 0.141

Preliminary conclusion I

For perfectly known breakpoints, time series lengths of 100 with 5 breaks, SNR = 1, and no mean trend introduced by the breaks the situation is:

The result after homogenization is slightly better than the original.

The mean trend is zero, the uncertainty before is: 0.265and afterwards: 0.141

Ratio q: (Improvement)


Improvement ratio (1/2)Ratio q depends on the SNR.

Upper panel:Doubled noise (SNR = ½ ) leads to doubled remaining trend error. The inserted trend error is unchanged.Doubled q

Lower panel:Doubled signal (SNR = 2) leads to doubled inserted trend error. The remaining trend error is unchanged.Halved q

Inserted and remaining errors are independent.The inserted error is determined by the break (signal) variance.The remaining error is determined by the noise variance.


Inserted

sx2 = 0.265

sy2 = 0.563

Rem

aini

ng

Inserted

Rem

aini

ng

SNR = ½

sx2 = 1.060

sy2 = 0.141

SNR = 2

q = 1.46 (0.73)

q = 0.365 (0.73)

Improvement ratio (2/2)

Further parameters (besides SNR) that may also affect q are: break and station number.

It shows:The ratio does mainly depend on break number and not on station number.

For 6 breaks the ratio is about 1.


Break number

Stat

ion

num

ber

0.5 1.0 1.5

Preliminary conclusion II Does the correction act neutrally if no correction is necessary?

Yes, depending on SNR,for SNR=1, break number=6, length=100 the data is neither upgraded nor downgraded.

But,the obtained homogenized data is mutually dependent.Standard statistical techniques (using data independency) cannot be applied.All variances are underestimated.


And IF there is a trend error?

To simulate this, a fourth signal is added:

The heights of the inhomogeneities are shifted by DI upward or downward depending on the middlemost year of the subperiod jm.

Due to boundary effects the mean introduced trend error is decreased from 1 (as naively expected) to 0.873 K/cty


Year

Mea

n in

sert

ed D

I

Non-zero mean trend error

The scatter is conserved, compared to zero mean trend error.

The data cloud is shifted as a whole to the right.

The uncertainty remains, but the mean trend error is well corrected.



Rem

aini

ng n

etw

ork-

mea

n tr

end

erro

r

xmean = 0.873ymean = 0.010

Including break position errors

Question:What happens, if the break positions have errors and are not perfectly known (as in reality).

Simulation:Scatter the correct positions by adding noise with standard deviation of 2 years.

Result:Only 80% of the trend error is corrected, 20% remains.



Rem

aini

ng n

etw

ork-

mea

n tr

end

erro

r

xmean = 0.873ymean = 0.161

Conclusion

If the original data contains no trend error (if the inhomogeneities have by chance no overall effect) the trend error for individual networks is (under realistic conditions) not improved.

However, mutually dependent data results from homogenization. This is a disadvantage.

Mean trend errors (due to inhomogeneities) are corrected perfectly, if the break positions are known perfectly.

For realistic position errors (s = 2 years) the trend error is only partly corrected, 20% remains.


Documents

Correction of spurious trends in climate series caused by inhomogeneities Ralf Lindau