Transcript
Page 1: Attractors  in Neurodynamical Systems

Attractors Attractors in Neurodynamical Systemsin Neurodynamical Systems

Włodzisław Duch, Włodzisław Duch, Krzysztof Dobosz Krzysztof Dobosz

Department of InformaticsDepartment of InformaticsNicolaus Copernicus UniversityNicolaus Copernicus University, , Toruń, PolandToruń, Poland

Google: W. DuchGoogle: W. Duch

ICNNICNN, , Hangzhou, Nov Hangzhou, Nov 20020099

Page 2: Attractors  in Neurodynamical Systems

MotivationMotivation• Neural respiratory rhythm generator (RRG): hundreds of Neural respiratory rhythm generator (RRG): hundreds of

neurons, what is the system doing?neurons, what is the system doing?• Analysis of multi-channel, non-stationary, time series data.Analysis of multi-channel, non-stationary, time series data.• Information is in the trajectories, but how to see in high-D? Information is in the trajectories, but how to see in high-D?

• Component-based analysis: ICA, NNMF, wavelets ...Component-based analysis: ICA, NNMF, wavelets ...• Time-frequency analysis, bumps ... Time-frequency analysis, bumps ... • Recurrence plots, state portraits: limited info about trajectories.Recurrence plots, state portraits: limited info about trajectories.

Fuzzy Symbolic Dynamics (FSD): visualize + understand.Fuzzy Symbolic Dynamics (FSD): visualize + understand.1.1. Understand FSD mappings using simulated data.Understand FSD mappings using simulated data.2.2. First looks at some real data.First looks at some real data.3.3. Examples from simulations of semantic word recognition.Examples from simulations of semantic word recognition.

Page 3: Attractors  in Neurodynamical Systems

Brain SpirographyBrain Spirography

Example of a pathological signal analysisExample of a pathological signal analysis

Page 4: Attractors  in Neurodynamical Systems

Recurrent plots and trajectoriesRecurrent plots and trajectoriesTrajectory of dynamical Trajectory of dynamical systemsystem (neural activities, av. rates): (neural activities, av. rates):

1..1..( ) { ( )}t N

i i nt x t x

Use time as indicator of minimal distance: Use time as indicator of minimal distance:

For discretized time steps binary matrix For discretized time steps binary matrix RRijij is obtained. is obtained.

Many measure of complexity and dynamical invariants may Many measure of complexity and dynamical invariants may be derived from RP: generalized entropies, correlation be derived from RP: generalized entropies, correlation dimensions, mutual information, redundancies, etc. dimensions, mutual information, redundancies, etc.

N. Marwan et al, Phys. Reports 438 (2007) 237-329. N. Marwan et al, Phys. Reports 438 (2007) 237-329.

Embedding of time series or mutidimensional trajectories.Embedding of time series or mutidimensional trajectories.

( , ' ; ) ( ) ( ')R t t x t x t

Page 5: Attractors  in Neurodynamical Systems

Recurrence plotsRecurrence plots• Unfold the trajectory at Unfold the trajectory at tt and show when it comes close and show when it comes close

to to xx((tt))..

Page 6: Attractors  in Neurodynamical Systems

Fuzzy Symbolic Dynamics (FSD)Fuzzy Symbolic Dynamics (FSD)Trajectory of dynamical Trajectory of dynamical systemsystem (neural activities, av. rates): (neural activities, av. rates):

1. Standardize data.1. Standardize data.

2. Find cluster centers (e.g. by k-means algorithm): 2. Find cluster centers (e.g. by k-means algorithm): 11, , 2 2 ......

3. Use non-linear mapping to reduce dimensionality:3. Use non-linear mapping to reduce dimensionality:

T 1( ; , ) expkk k k k ky t x x

Localized membership functions: Localized membership functions:

sharp indicator functions => symbolic dynamics; strings.sharp indicator functions => symbolic dynamics; strings.

soft membership functions => fuzzy symbolic dynamics, soft membership functions => fuzzy symbolic dynamics, dimensionality reduction => visualization.dimensionality reduction => visualization.

0 0 0( ( ), ) ( ) exp ( )S t t t x x x x x x

Page 7: Attractors  in Neurodynamical Systems

Model, radial/linear sourcesModel, radial/linear sources

Sources generate waves on a gridSources generate waves on a grid

( ; ) coslij l l ijF t t p k p

( ; ) cos ,lij l l l i jR t t k r x y p

( ; ) , ,l lij ij ij

l l

A t F t p R t p p

Flat waveFlat wave

Radial wave

Relatively simple patterns arise, but slow sampling shows Relatively simple patterns arise, but slow sampling shows numerical artifacts.numerical artifacts.

Ex: one and two radial waves.Ex: one and two radial waves.

Page 8: Attractors  in Neurodynamical Systems

Radial + plane wavesRadial + plane waves

Radial sources are turned on and off, 5 events+transients.Radial sources are turned on and off, 5 events+transients.

Page 9: Attractors  in Neurodynamical Systems

Respiratory Rhythm GeneratorRespiratory Rhythm Generator

3 layers, spiking neurons, output layer with 50 neurons3 layers, spiking neurons, output layer with 50 neurons

Page 10: Attractors  in Neurodynamical Systems

Sensitive differences?Sensitive differences?

Page 11: Attractors  in Neurodynamical Systems

Sensitive differences?Sensitive differences?

Page 12: Attractors  in Neurodynamical Systems

FSD developmentFSD development

Optimization of parameters of membership functions to see more Optimization of parameters of membership functions to see more structure from the point of view of relevant task.structure from the point of view of relevant task.

Learning: supervised clustering, projection pursuit based on quality Learning: supervised clustering, projection pursuit based on quality of clusters => projection on interesting directions.of clusters => projection on interesting directions.

Measures to characterize dynamics: position and size of basins of Measures to characterize dynamics: position and size of basins of attractors, transition probabilities, types of oscillations around attractors, transition probabilities, types of oscillations around each attractor (follow theory of recurrent plots for more).each attractor (follow theory of recurrent plots for more).

Visualization in 3D and higher (lattice projections etc).Visualization in 3D and higher (lattice projections etc).

Tests on model data and on the real data. Tests on model data and on the real data.

Page 13: Attractors  in Neurodynamical Systems

BCI EEG exampleBCI EEG example Data from two electrodes, BCI IIIaData from two electrodes, BCI IIIa

Page 14: Attractors  in Neurodynamical Systems

Alcoholics vs. controlsAlcoholics vs. controls

Colors: from blue at the beginning of the sequence, to red at the end.Colors: from blue at the beginning of the sequence, to red at the end.

Left: normal subject; right: alcoholic; task: two matched stimuli, Left: normal subject; right: alcoholic; task: two matched stimuli, 64 channels (3 after PP), 256 Hz sampling, 64 channels (3 after PP), 256 Hz sampling, 1 1 sec, 10 trialssec, 10 trials; single st alc; single st alc..

Page 15: Attractors  in Neurodynamical Systems

Model Model of readingof reading

Learning: mapping one of the Learning: mapping one of the 3 3 layers to the other two.layers to the other two.

Fluctuations around final configuration = attractors representing concepts.Fluctuations around final configuration = attractors representing concepts.

How to see properties of their basins, their relations?How to see properties of their basins, their relations?

Emergent neural simulator:Emergent neural simulator:Aisa, B., Mingus, B., and O'Reilly, R. Aisa, B., Mingus, B., and O'Reilly, R. The emergent neural modeling The emergent neural modeling system. Neural Networks, system. Neural Networks,

21, 1045-1212, 2008. 21, 1045-1212, 2008.

3-layer model of reading: 3-layer model of reading: orthography, phonology, semantics, orthography, phonology, semantics, or distribution of activity using 140 or distribution of activity using 140 microfeatures of concepts. microfeatures of concepts. Hidden layers in between. Hidden layers in between.

Page 16: Attractors  in Neurodynamical Systems

AttractorsAttractors

FSD representation of 140-dim. trajectories in 2 or 3 dimensions. FSD representation of 140-dim. trajectories in 2 or 3 dimensions.

Attractor landscape changes in time due to neuron accommodation. Attractor landscape changes in time due to neuron accommodation.

Page 17: Attractors  in Neurodynamical Systems

2D attractors for words 2D attractors for words Dobosz K, Duch W, Fuzzy Symbolic Dynamics for Neurodynamical Systems. Dobosz K, Duch W, Fuzzy Symbolic Dynamics for Neurodynamical Systems. Neural Networks (in print, 2009). Same 8 words, more synaptic noise.Neural Networks (in print, 2009). Same 8 words, more synaptic noise.

Page 18: Attractors  in Neurodynamical Systems

Depth of attractor basinsDepth of attractor basinsVariance around the center of a cluster grows with synaptic noise; for narrow Variance around the center of a cluster grows with synaptic noise; for narrow and deep attractors it will grow slowly, but for wide basins it will grow fast. and deep attractors it will grow slowly, but for wide basins it will grow fast. Jumping out of the attractor basin reduces the variance due to inhibition of Jumping out of the attractor basin reduces the variance due to inhibition of desynchronized neurons. desynchronized neurons.

Page 19: Attractors  in Neurodynamical Systems

3D attractors for words3D attractors for wordsNon-linear visualization of activity of Non-linear visualization of activity of the semantic layer with 140 units for the semantic layer with 140 units for the model of reading that includes the model of reading that includes phonological, orthographic and phonological, orthographic and semantic layers + hidden layers. semantic layers + hidden layers.

Cost /wage, hind/deer have semantic Cost /wage, hind/deer have semantic associations, attractors are close to associations, attractors are close to each other, but without neuron each other, but without neuron accommodation attractor basins are accommodation attractor basins are tight and narrow, poor generalization tight and narrow, poor generalization expected.expected.

Training with more variance in phonological and written form of words may Training with more variance in phonological and written form of words may help to increase attractor basins and improve generalization. help to increase attractor basins and improve generalization.

Page 20: Attractors  in Neurodynamical Systems

Connectivity effectsConnectivity effects

Same situation but recurrent Same situation but recurrent connections within layers are connections within layers are stronger, fewer but larger stronger, fewer but larger attractors are reached, attractors are reached, more time is spent in each more time is spent in each attractor. attractor.

With small synaptic noise With small synaptic noise (var=0.02) the network starts from (var=0.02) the network starts from reaching an attractor and moves to reaching an attractor and moves to creates “chain of thoughts”.creates “chain of thoughts”.

Page 21: Attractors  in Neurodynamical Systems

Inhibition effectsInhibition effects

Prompting the system Prompting the system with single word and with single word and following noisy following noisy dynamics, not all dynamics, not all attractors are real attractors are real words. words.

Increasing Increasing ggii from 0.9 to 1.1 from 0.9 to 1.1 reduces the attractor reduces the attractor basins and basins and simplifies trajectories.simplifies trajectories.

Page 22: Attractors  in Neurodynamical Systems

ExplorationExploration

Like in molecular dynamics, long Like in molecular dynamics, long time is needed to explore various time is needed to explore various potential attractors – depending potential attractors – depending on priming (previous dynamics or on priming (previous dynamics or context) and chance.context) and chance.

Same parameters but different Same parameters but different runs: each time a single word is runs: each time a single word is presented and dynamics run presented and dynamics run exploring different attractors.exploring different attractors.

Page 23: Attractors  in Neurodynamical Systems

Neurons and dynamicsNeurons and dynamicsTrajectories show spontaneous shifts of attention. Trajectories show spontaneous shifts of attention. o Attention shifts may be impaired due to the deep and narrow attractor Attention shifts may be impaired due to the deep and narrow attractor

basins that entrap dynamics – dysfunction of leak channels (~15 types)?basins that entrap dynamics – dysfunction of leak channels (~15 types)?

In memory models overspecific memories are created (as in ASD), unusual In memory models overspecific memories are created (as in ASD), unusual attention to details, the inability to generalize visual and other stimuli.attention to details, the inability to generalize visual and other stimuli.

o Accommodation: voltage-dependent KAccommodation: voltage-dependent K++ channels (~40 types) do not channels (~40 types) do not decrease depolarization in a normal way, attractors do not shrink.decrease depolarization in a normal way, attractors do not shrink.

This should slow down attention shifts and reduce jumps to unrelated This should slow down attention shifts and reduce jumps to unrelated thoughts or topics (in comparison to average person). thoughts or topics (in comparison to average person). Neural fatigue temporarily turns some attractors off, making all attractors Neural fatigue temporarily turns some attractors off, making all attractors that code significantly overlapping concepts inaccessible. that code significantly overlapping concepts inaccessible.

This is truly dynamic picture: attractor landscape changes in time! This is truly dynamic picture: attractor landscape changes in time! What behavioral changes are expected depending on connectivity, inhibition, What behavioral changes are expected depending on connectivity, inhibition,

accommodation dynamics, leak currents, etc? accommodation dynamics, leak currents, etc?

Page 24: Attractors  in Neurodynamical Systems

What can we learn?What can we learn?o Visualization should give insight into general behavior of neurodynamical Visualization should give insight into general behavior of neurodynamical

systems, measure of complexity and dynamical invariants may be derived systems, measure of complexity and dynamical invariants may be derived along the lines of recurrence plots.along the lines of recurrence plots.

o How many attractors can be identified? How many attractors can be identified? o Where does the system spends most of its time?Where does the system spends most of its time?o Where is the trajectory most of the time? Where is the trajectory most of the time? o What are the properties of basins of attractors (size, depths, time spend)?What are the properties of basins of attractors (size, depths, time spend)?o What are the probabilities of transition between them (distances )?What are the probabilities of transition between them (distances )?o How fast the transition occurs? How fast the transition occurs? o What type of oscillations occur around the attractors? Chaos? What type of oscillations occur around the attractors? Chaos? o FSD shows global mapping of the whole trajectory (do we want that?).FSD shows global mapping of the whole trajectory (do we want that?).

o Different conditions more easily distinguished and interpreted than in Different conditions more easily distinguished and interpreted than in recurrence plots, potentially useful in classification and diagnosis.recurrence plots, potentially useful in classification and diagnosis.

Page 25: Attractors  in Neurodynamical Systems

Future plansFuture plans

• Relations between FSD, symbolic dynamics, Relations between FSD, symbolic dynamics, and recurrence plots.and recurrence plots.

• Simulated EEG models to understand how to interpret the FSD plots. Simulated EEG models to understand how to interpret the FSD plots. • Other visualization methods: MDS, LLE, Isomap, LTSA, diffusion map … Other visualization methods: MDS, LLE, Isomap, LTSA, diffusion map … • Effects of various component-based transformations: PCA, ICA, NNMF ...Effects of various component-based transformations: PCA, ICA, NNMF ...• Supervised learning of membership function parameters to find interesting Supervised learning of membership function parameters to find interesting

structures in low-dimensional maps: structures in low-dimensional maps: adding projection pursuit to find interesting views; adding projection pursuit to find interesting views; projection pursuit in space and time to identify interesting segments.projection pursuit in space and time to identify interesting segments.

• Combining projection pursuit with time-frequency analysis and FSD for EEG Combining projection pursuit with time-frequency analysis and FSD for EEG analysis. analysis.

• Systematic investigation of parameters of neurodynamics on basins of Systematic investigation of parameters of neurodynamics on basins of attractors. attractors.

• BCI and other applications + many other things … BCI and other applications + many other things …

Page 26: Attractors  in Neurodynamical Systems

Thank Thank youyoufor for

lending lending your your ears ears

......

Google: W. Duch => Papers & presentationsSee also http:www.e-nns.org


Recommended