S. Srinivasan, S. Prasad, S. Patil, G. Lazarou and J. Picone Intelligent Electronic Systems Center for Advanced Vehicular Systems Mississippi State University

S. Srinivasan, S. Prasad, S. Patil, G. Lazarou and J. Picone

Intelligent Electronic SystemsCenter for Advanced Vehicular Systems

Mississippi State University

URL: http://www.cavs.msstate.edu/hse/ies/publications/conferences/ieee_secon/2006/lyapunov_exponents/

ESTIMATION OF LYAPUNOV SPECTRA

FROM A TIME SERIES

http://www.cavs.msstate.edu/hse/ies/publications/conferences/ieee_secon/2006/lyapunov_exponents/

Page 2 of 24Estimation of Lyapunov Spectra from a Time Series

• Analysis of chaotic signals:

• Reconstruction a phase-space from a scalar observable

• Lyapunov exponents as a tool to analyze chaos

• Lyapunov spectra of chaotic and non-chaotic time series

• Optimize parameters of Lyapunov spectra estimation

Motivation


Definitions

• A deterministic signal or system: every event is the result of preceding events and actions; hence predictable completely

• Stochastic noise: signal that is not deterministic, i.e., inherently unpredictable

• A chaotic signal or system: sensitive to initial conditions (Butterfly Effect)

• Chaos says: predictability holds only in principle, hence chaotic signals are also called deterministic noise.

• Dimension of a system: number degrees of freedom possessed by the system

• Deterministic Chaos or Stochastic Noise?

Both have continuous power spectra (and not easily distinguishable)

Noise is infinite-dimensional.

Chaotic signals are finite dimensional, but dimension no longer associated with number of independent frequencies, but a statistical feature related to both temporal evolution and geometric aspect (self-similar structure of the attractor)


Power Spectrum of a Lorentz Signal

• Power spectra of chaotic signals are continuous, though the system is finite dimensional. For example, the power spectrum of a 3-dimensional chaotic Lorentz signal is shown below. Stochastic systems have similar spectra even though they are infinite dimensional.


Attractors for Dynamical Systems

• System Attractor: Trajectories approach a limit with increasing time, irrespective of the initial conditions within a region

• Basin of Attraction: Set of initial conditions converging to a particular attractor

• Attractors: Non-chaotic (point, limit cycle or torus),or chaotic (strange attactors)

• Example: point and limit cycle attractors of a logistic map (a discrete nonlinear chaotic map)


Strange Attractors

• Strange Attractors: attractors whose shapes are neither points nor limit cycles. They typically have a fractal structure (i.e., they have dimensions that are not integers but fractional)

• Example: a Lorentz system with parameters 0.40.40,0.16 br and


Characterizing Chaos

• Exploit geometrical (self-similar structure) aspects of an attractor or the temporal evolution for system characterization

• Geometry of a Strange Attractor:

Most strange attractors show a similar structure at various scales, i.e., parts are similar to the whole.

Fractal dimensions can be used to quantify this self-similarity.

e.g., Hausdorff, correlation dimensions.

• Temporal Aspect of Chaos:

Characteristic exponents or Lyapunov Exponents (LE’s) - captures rate of divergence (or convergence) of nearby trajectories;

Also Correlation Entropy captures similar information.

• Any characterization presupposes that phase-space is available.

What if only one scalar time series measurement of the system (and not its actual phase space) is available?


Reconstructed Phase Space (RPS): Embedding

• Embedding: A mapping from an one-dimensional signal to an m-dimensional signal

• Taken’s Theorem:

Can reconstruct a phase space “equivalent” to the original phase space by embedding with m ≥ 2d+1 (d is the system dimension)

• Embedding Dimension: a theoretically sufficient bound; in practice, embedding with a smaller dimension is adequate.

• Equivalence:

means the system invariants characterizing the attractor are the same

does not mean reconstructed phase space (RPS) is exactly the same as original phase space

• RPS Construction: techniques include differential embedding, integral embedding, time delay embedding, and SVD embedding


Reconstructed Phase Space (RPS): Time Delay Embedding

• Uses delayed copies of the original time series as components of RPS to form a matrix

m: embedding dimension, : delay parameter

• Each row of the matrix is a point in the RPS

)1(222

)1(111

)1(0

m

m

m

xxx

xxx

xxx

X


Reconstructed Phase Space (RPS)

Time Delay Embedding of a Lorentz time series


Reconstructed Phase Space (RPS): Time Delay Embedding

• Setting very small delay value: leads to highly correlated vector elements, concentrated around the diagonal in embedding space. Structure perpendicular to the diagonal not captured adequately.

• Setting very large delay value: leads elements of the vector to behave as if they are independent. Evolutionary information in the system is lost.

• Quantitative tools for fixing delay: plots of autocorrelation and auto-mutual information are useful guides.

• Advantages: easy to compute; the attractor structure is not distorted since no extra processing is done on it.

• Disadvantages: choice of delay parameter value is not obvious; leads to poor RPS in presence of noise.


Reconstructed Phase Space (RPS): SVD-based Embedding

• Works in two stages:

1.Delay embed, with one sample delay, to a dimension larger than twice the actual embedding dimension

2.Reduce this matrix using SVD to finally have number of columns equal to embedding dimension.

(SVD-based matrix reduction is done by projecting each row onto only the first few eigenvectors and then reconstructing it to a lower-dimensional space)

• SVD window size: dimension of time delayed embedded matrix over which SVD operates

• Advantages: No delay parameter value to be set; more robust to noise due to SVD stage

• Disadvantages: Noise reducing property of SVD may also distort the attractor properties


Reconstructed Phase Space (RPS): Reconstruction

Attractor reconstruction using SVD embedding (for a Lorentz system)


Lyapunov Exponents

• Quantifies separation in time between trajectories, assuming rate of growth (or decay) is exponential in time, as:

where J is the Jacobian matrix at point p.

• Captures sensitivity to initial conditions.

• Analyzes separation in time of two trajectories with close initial points

where is the system’s evolution function.

)0()()0()( xfdx

dxtx N

)J(p)eigln(1

limn

0p

nn

i

f

f


Lyapunov Exponents – Some Properties

• m-dimensional system has m LE’s

• LE is a measure averaged over the whole attractor

• Sum of first k LE’s: rate of growth of k-dimensional Euclidean volume element

• Bounded attractor: Sum of all LEs equals zero (conservative) or negative (dissipative)

• Zero exponents indicate periodic attractor (limit cycle) or a flow

• Negative exponents pull points in the basin of attraction to the attractor

• Positive exponents indicate divergence: signature for existence of chaos


Lyapunov Exponents: Computation

1.Embed time series to form RPS matrix. Rows represent points in phase space

2.Take first point as center

3.Form neighborhood matrix, each row obtained by subtracting a neighbor from the centre

4.Find evolution of each neighbor and form the evolved neighborhood matrix by subtracting each evolved neighbor from the evolved centre

5.Compute trajectory matrix at the center by multiplying pseudo-inverse of neighborhood matrix with evolved neighborhood matrix

6.Advance center to a new point and go to step 3, averaging the trajectory matrix in each iteration

The LEs are given by the average of the eigenvalues from each R matrix. Direct averaging has numerical issues, hence an iterative QR decomposition method (treppen-iteration) is used.


Lyapunov Exponents: Computation Flowchart

Embed the data

Locate nearest points

Evolve “a” steps

Calculate trajectory martix

Form neighborhood

Locate nearest points

Perform QR decomposition

oLocate nearest points

Calculate exponents from R

Evolve center

Form neighborhood

Input time series


Experimental Design

• Three systems tested : two chaotic (Lorentz and Rossler) and one non-chaotic (sine signal)

• Two test conditions: clean and noisy (10 dB white noise)

• Lorentz system:

• Parameters:

• Expected LEs: (+1.37, 0, -22.37)

• Rossler system:

• Parameters: a = 0.15, b = 0.2, c = 10

• Expected LEs: (0.090, 0.00, -9.8)

• Sine Signal:

• Parameters: Freq=1Hz, Samp freq=16Hz, Amp=1

• Expected LEs: (0.00, 0.00, -1.85)

0.4 and 0.40,0.16 br


Experimental Design

• Experiments performed to optimize parameters of estimation algorithm

• 30,000 points were generated for each series in both the conditions

• 5,000 iterations (or the number of evolution steps) were used for averaging using QR treppen-iteration

• Variation of LEs with SVD window size and number of nearest neighbors

• Varied number of neighbors with SVD window size: 15 for clean data; 50 for noisy data

• Varied SVD window size with number of neighbors: 15 for clean data; 50 for noisy data


Experimental Results

Lyapunov Exponents (LEs) for a Lorentz System

• For clean data: Positive and zero exponents are almost constant at the expected values

• For noisy data: Positive and zero exponents converge to the expected value for SVD window size about 50 and number of neighbors also about 50

• Negative LE estimation: not reliable



Lyapunov Exponents (LEs) for a Rossler System


• For noisy data: Positive and zero exponents converge to the expected value for SVD window size about 60 and number of neighbors also about 50




Lyapunov Exponents (LEs) for Sine Signal


• For clean data: Positive and zero exponents converge to the expected value for SVD window size about 40 and number of neighbors also about 30



Summary and Future Work

• LEs are useful in quantifying how chaotic a system is.

• SVD embedding helps reconstructing phase spaces in noisy conditions.

• Parameters of the LE computation algorithm are optimized experimentally to get reliable estimates.

• Both the positive and zero LE’s are estimated near the actual values using optimized parameters.

• Negative LE estimation is unreliable (but this is of little concern in chaotic systems).

• The code for LE estimation is publicly available.

• Our future work will be to apply Lyapunov exponents to model nonlinearities in speech for better automatic speech recognition.


Resources

• Pattern Recognition Applet: compare popular linear and nonlinear algorithms on standard or custom data sets

• Speech Recognition Toolkits: a state of the art ASR toolkit for testing the efficacy of these algorithms on recognition tasks

• Foundation Classes: generic C++ implementations of many popular statistical modeling approaches


References

• J. P. Eckmann and D. Ruelle, “Ergodic Theory of Chaos and Strange Attractors,” Reviews of Modern Physics, vol. 57, pp. 617‑656, July 1985.

• M. Banbrook, “Nonlinear analysis of speech from a synthesis perspective,” PhD Thesis, The University of Edinburgh, Edinburgh, UK, 1996.

• E. Ott, T. Sauer, J. A. Yorke, Coping with chaos, Wiley Interscience, New York, New York, USA, 1994.

• M. Sano and Y. Sawada, “Measurement of the Lyapunov Spectrum from a Chaotic Time Series,” Physical Review Letters, vol. 55, pp. 1082-1085, 1985.

• G. Ushaw, “Sigma delta modulation of a chaotic signal,” PhD Thesis, The University of Edinburgh, Edinburgh, UK, 1996.

Documents

S. Srinivasan, S. Prasad, S. Patil, G. Lazarou and J. Picone Intelligent Electronic Systems Center for Advanced Vehicular Systems Mississippi State University