Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
E2C2-GIACS Advanced School on
‘Extreme Events: Nonlinear Dynamics and Time Series Analysis’
Smooth Time Series - Applications
Dmitri Kondrashov
University of California, Los Angeles
Joint work with M. Ghil, and many others
http://www.atmos.ucla.edu/tcd
1/32
Spectral Analysis
S(f) ~ f – p + poles
i.e. linear in log-log coordinates
For a 1st-order Markov process or “red noise” p = 2
“Pink” noise, p = 1 (1/f , flicker noise)
“White” noise, p = 0
Main challenge for short and noisy geophysical time series to distinguish between poles and (red) noise.
Tradeoff for spectral methods: resolution (smoothing, windowing) vs. spurious peaks & power leakage.
6/32
x = −ω2x vs. x = −λx
Synthetic example (Task & Prize!!!)
7/32
Datax101
0.11
0.21
0.00
-0.11
-0.21
1.0 40.8 80.6 120.4 160.2 200.0
Q: Is there a periodicity and what is its period?
A: What is the underlying noise “null hypothesis”?
Classical Spectral Methods
AR(1)95%
Blakman-Tukey FFT (window 80)
Frequency
0.00 0.10 0.20 0.30 0.40 0.50
10-1
10-0
101
AR(1)95%
Blakman-Tukey FFT (window 80)
Frequency
0.00 0.10 0.20 0.30 0.40 0.50
10-1
10-0
101
AR(1)95%
Blakman-Tukey FFT (window 120)
Frequency
0.00 0.10 0.20 0.30 0.40 0.50
10-1
10-0
101
AR(1)95%
Blakman-Tukey FFT (window 120)
Frequency
0.00 0.10 0.20 0.30 0.40 0.50
10-1
10-0
101
MEMMaximum Entropy Method (40)
Frequency
0.00 0.10 0.20 0.30 0.40 0.50
10-1
10-0
101
8/32
MEMMaximum Entropy Method (10)
Frequency
0.00 0.10 0.20 0.30 0.40 0.50
100
101
Advanced Spectral Methods
10/32
••
••
•••
•••
•
•••
••
•••
••
••
••
••••
••
••••••••
Singular Spectrum Analysis
Frequency
0.00 0.10 0.20 0.30 0.40 0.49
10-1
10-0
101
Multi-taper method
Frequency
AR(1) 99% 95% 90%
0.00 0.10 0.20 0.30 0.40 0.50
10-1
10-0
101
- Singular spectrum analysis (SSA)and Multi-taper method (MTM).
- detection of periodic signals: phase and amplitude modulation, intermittent behavior, large noise.
- use data-adaptive orthogonal basis in frequency domain (MTM) and time domain (SSA).
- significance tests for spectral peaks.
Right answer!
9/32
Signal
Synthetic Datax101
0.11
0.21
0.00
-0.11
-0.21
1.0 40.8 80.6 120.4 160.2 200.0
SSA Power Spectra & Reconstruction
A. Transform pair:
For given window M, ek’’s are adaptive filters (empirical orthogonal functions)
the ak’’s are filtered time series, principal components in time domain.
B. Power spectra
C. Reconstruction
in particular:
ak(t) =M∑
s=1
X(t + s)ek(s), ak(t)− PC
X(t + s) =M∑
k=1
ak(t)ek(s), ek(s)− EOF
XK(t) =1M
∑
k∈K
M∑
s=1
ak(t− s)ek(s);
12/32
K = {1, 2, ....., S} or K = {k} or K = {l, l + 1;λl ≈ λl+1}
SX(f) =M∑
k=1
Sk(f); Sk(f) = Rk(s); Rk(s) ≈1
T
∫ T
0
ak(t)ak(t + s)dt
SSA of Southern Oscillation Index (ENSO)
•Powerful noise filter: Break in slope of SSA spectrum distinguishes “significant” from “noise” EOFs
14/32
•Formal Monte-Carlo test identifies 4-yr and 2-yr ENSO oscillatory modes (SSA pairs). A window size of M = 60 is enough to “resolve” these modes in a monthly SOI time series.
SSA Forecast (ENSO)
Filter “signal” and forecast with AR(M) model.
Cross-validation to find optimum number of “signal” components and error bars of the forecast.
Correlations are both advantage and limitations of empirical models.
15/32
Real-time Plume of Climate Forecasts (ENSO)
16/32
IRI Multi-model forecast of Nino-34 index
UCLA-TCD: Kondrashov, D., S. Kravtsov, A. W. Robertson, and M. Ghil (2005), A hierarchy of data-based ENSO models, J. Climate, 18, 4425–4444. 495
!"" "#$ "%& %&' &'( '() ()" )"* "*! *!% !%! %!"!+,-
!+
!.,-
!.
!/,-
/
/,-
.
.,-
+
+,-
0
OBS FORECAST
(1('0,23&&43%5678$9:6;<
!6=>$3*6?>@8ABA36C3D(&'3C?673Aug 2007
IRI
+//E +//F
(%&%3G!%'
(;DH3;*&
"!%
&;I1HH&
J)D'
%K&LH'%!%
D;!M*
KN!'
N!%3&(K
D;O%!L!'!
;'J%3%('!
!>B*I%(;D
"H(!*I;G;
;H;3!IN'P
;);3J1!
;H;3;%
;H;3;;%
;&K3;J1HI
KQ;3((D4
*&K3IDGI
K;J%!4;)
)9587R@8$3!6=>$S
&B8BRABR@8$3!6=>$S
Dealing with
Missing
Data
17/32
Historical records are full of “gaps”....
Annual maxima and minima of the water level at the nilometer on Rodah Island, Cairo. 18/32
... nowdays on Earth...
SST (AMSR-E), daily 2x2, June 2002 – February 2007: 38.2% of missing points
Wind (QuikSCAT), daily 2x2, July 1999 -- February 2007:17.2% of missing points
Gaps: satellite coverage, precipitation and clouds.
20/32
23/32
1. Choose window M and set K=1. Flag fraction of dataset X(t)(t=1:N) as “missing” for cross-validation.
D =
X(1) X(2) . . X(M)X(2) X(3) . . X(M + 1)
. . . . .
X(N ′− 1) . . . X(N − 1)
X(N ′) X(N ′ + 1) . . X(N)
CX =1
N ′D
tD;CXEk = λkEk
Ak(t) =∑M
j=1X(t + j − 1)Ek(j)
RK(t) = 1
Mt
∑k∈K
∑Ut
j=LtAk(t − j + 1)Ek(j);
4. If convergence for missing points, K = K +1. Check cross-validation error, and Go to Step 2 if necessary.
2. Update mean and covariance, find leading K EOFs
3. Reconstruct missing points using K EOFs
Utilize (spatio) temporal correlations to iteratively compute self-consistent lag-covariance matrix => can be applied to very gappy data.
Follows expectation maximization (EM) procedure for finding maximum likelihood estimates of mean and covariance matrix.
A few K leading EOFs correspond to the “smooth” modes, while the rest is noise.
Provides both spectral analysis and estimates of missing data.
SSA gap-filling
D. Kondrashov and M. Ghil, 2006: Spatio-temporal filling of missing points in geophysical data sets,Nonl. Proc. Geophys., 13, 151-159.
Synthetic I: Gaps in Oscillatory Signal
• Very good gap filling for smooth modulation; OK for sudden modulation.
0 50 100 150 200!5
0
5a)SSA gap filling in [20:40] range
0 2 4 6 8 10
0.2
0.4
No. of modes
CVL
Erro
r
Optimum window M and number of modes101520
0 50 100 150 200!5
0
5b)SSA gap filling in [120:140] range
0 2 4 6 8
0.2
0.4
No. of modesCV
L Er
ror
Optimum window M and number of modes
152025
0 0.1 0.2 0.3 0.4 0.5
100
SSA spectrum, M=10
Frequency0 0.1 0.2 0.3 0.4 0.5
100
SSA spectrum, M=20
Frequency
24/32
Synthetic II: Gaps in Oscillatory Signal + Noise
x(t) = sin( 2π300 t) ∗ cos( 2π
40 t + π2 sin 2π
120 t)25/32
! "!! #!! $!!!%
!
%&'()*++*,-(-&.(/
0! "!! #!! $!!
!%
!
%
0
1'()*++*,-(-&.(//
2! %!! %2!
!"!"
0
3'()*++*,-(-&.(/
"!! "2! 4!! 42!
!"!"
0
5'()*++*,-(-&.(//
" # $ 6 %!
%
%7!2
897(9:(;95<=
<'(>?9==!@&+*5&0*9,(ABC(<??9?
! !7!" !7!#
%!!
:'(CCD(=.<30?E;F(BG"!!
)?<HE<,3I
Synthetic III: Synthetic gaps in SST data
1950-2004 IRI monthly SST dataset (10°x10°, 660x237 grid points) see improvement with MSSA; “random” pattern favors small M!Error is smaller in CE Pacific Sector where “signal” (ENSO) mode is dominant!
Synthetic IV: multivariate example (Prize!!!)
22/32
-2.00e+00
2.00e+00Dataset with 50% of missing data
Time
Spac
e
x102
0.01
0.07
0.14
0.20
1.0 20.8 40.6 60.4 80.2 100.0
-2.00e+00
2.00e+00Estimated smooth component
Time
Sp
ace
x102
0.01
0.07
0.14
0.20
1.0 20.8 40.6 60.4 80.2 100.0
-2.00e+00
2.00e+00Original noisy dataset
Time
Sp
ace
x102
0.01
0.07
0.14
0.20
1.0 20.8 40.6 60.4 80.2 100.0
-2.00e+00
2.00e+00Smooth component of full dataset
Time
Spac
e
x102
0.01
0.07
0.14
0.20
1.0 20.8 40.6 60.4 80.2 100.0
Filled-in Southern Ocean data
Gap-filling needs to respect physical limits
29/32
•Freeware ported to Sun, Dec, SGI, Linux, and Mac OS X: self-contained binary (~2-5Mb) depending on the Unix platform. •http://www.atmos.ucla.edu/tcd/ssa/ •Needs external graphics package: Grace (free, default) is a part of
standard Linux installation, may need compiling for other OS; IDL ($$)•Includes Blackman-Tukey FFT, Maximum Entropy Method, Multi-
Taper Method (MTM), SSA and M-SSA. •Spectral estimation, decomposition, reconstruction, gap-filling•Significance tests of “oscillatory modes” vs. “noise.”
30/32
SSA-MTM Toolkit
• Data management with named vectors & matrices.
• Default values. 31/32
SSA-MTM Toolkit (cont’d)
•Project files•SSA Forecasts•Automated tasks•Built-in plots•Animations (QuickTime)•Automation (Automator)•www.spectraworks.com
31/32
Mac OS X: kSpectra Toolkit
• Ghil M., R. M. Allen, M. D. Dettinger, K. Ide, D. Kondrashov, M. E. Mann, A. Robertson, A. Saunders, Y. Tian, F. Varadi, and P. Yiou, 2002: "Advanced spectral methods for climatic time series," Rev. Geophys.,40(1), pp. 3.1-3.41, 10.1029/2000RG000092.
• D. Kondrashov and M. Ghil, 2006: Spatio-temporal filling of missing points in geophysical data sets,Nonl. Proc. Geophys., 13, 151-159.
• more at http://www.atmos.ucla.edu/tcd/ssa.
• Computer Lab: SOI (ENSO), “small signal”, gap-filling, multivariate example (time permitting)
32/32
END