Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory

Complex networks, synchronization and

cooperative behaviour

Johan Suykens

KU Leuven, ESAT-SCD/SISTAKasteelpark Arenberg 10

B-3001 Leuven (Heverlee), BelgiumEmail: [email protected]

http://www.esat.kuleuven.be/scd/

VUB Leerstoel 2012-2013 - Oct. 31 2012

Complex networks, synchronization and cooperative behaviour

Introduction

http://www.youtube.com”synchronization of metronomes”

(a modern version of the synchronization of two pendulum clocks observedby Christiaan Huygens, 1665)

Complex networks, synchronization and cooperative behaviour 1

Overview

• Chaotic systems synchronization, Lur’e systems

• Cluster synchronization and community detection in complex networks

• Optimization using coupled local minimizers, cooperative behaviour


Circuits and systems: Chua’s circuit

+ +

− −

vC2C2 C1

iL

LvC1

gNR(vC1

)

NR

G

Ga

Gb

Ga

−EE

gNR(vC1

)

vC1

Chua’s circuit [Chua et al., 1986]: in dimensionless form

x = α(y − x − f(x))y = x − y + z

z = −βy

where

f(x) = m1x +1

2(m0 − m1) (|x + 1| − |x − 1|)

(depending on α, β: bistability, limit cycles, chaos)


Bifurcation to Chaos

(vC1, vC2)-plane:

Power spectrum vC1:

−→ birth of the double scroll attractor −→


Lur’e system

L

L(s)

N

σ(·)

m(t) = 0

u

y(t)++

k1

k2

σ(y)

y

• Lur’e system:

x = Ax + Bu

y = Cx

u = σ(y)→ x = Ax + Bσ(Cx)

where x ∈ Rn and σ(·) : R

h → Rh satisfies a sector condition.

• Chua’s circuit: h = 1


More hidden units

• Multi-stability & Multi-scroll chaos:Extend the nonlinearity andcreate additional equilibrium points[Suykens & Vandewalle, 1991; Arena, 1996; Yalcin, 2001; Lu, 2006]

• Multilayer neural networks are universal approximators [Hornik, 1989]

(Chua’s circuit has 1 hidden unit (h = 1), more hidden units for multi-scrolls)


A gallery of multi-scroll attractors

[Suykens & Vandewalle, 1991; Yalcin et al., 2001]


Lur’e systems: examples

• Lur’e system:

x = Ax + Bu

y = Cx

u = σ(y)→ x = Ax + Bσ(Cx)

• Many examples of Lur’e systems in different areas:- Recurrent neural networks (Hopfield network: A = −I, C = I) [Hopfield, 1985]

- Cellular neural networks (sparse and structured matrices A, B, C) [Chua, 1988]

- Actuator saturation in control systems

- Chua’s circuit, multi-scroll circuits

- Arrays of coupled networks

- Genetic oscillator models

L

L(s)

N

σ(·)

m(t) = 0

u

y(t)++

k1

k2

σ(y)

y


Genetic oscillators

A general genetic oscillator form [Li, Chen, Aihara, 2006]:

x(t) = Ax(t) +

l∑

i=1

Bifi(x(t))

where

• x(t) ∈ Rn: concentrations of proteins, RNAs, chemical complexes

• fi(x(t)) = [fi1(x1(t)); ...; fin(xn(t))]: regulatory function (monotonicallyincreasing or decreasing: e.g. Michaelis-Menten or Hill form)

Examples: Goodwin model, repressilator, toggle switch, circadian oscillators


Stability analysis and LMIs (1)

• Linear system:x = Ax

Quadratic Lyapunov function:

V = xTPx, P = P T > 0

• Stability analysis:

V = xTPx + xTP x = xT (ATP + PA)x < 0

Global asymptotic stability for

ATP + PA < 0

Linear matrix inequality (LMI) for a given matrix A [Boyd et al., 1994]


Stability analysis and LMIs (2)

• Lur’e system:x = Ax + Bσ(Cx)

Try e.g. a quadratic Lyapunov function (leading to a sufficient stabilitycondition):

V = xTPx, P = P T > 0

• Stability analysis: exploit the fact that σ belongs to sector [0, k]

V = xTPx + xTP x

≤ xTPx + xTP x−∑

i 2λiσi(σi − kcTi x) = [xTσT ]Z

[

x

σ

]

If

Z =

[

ATP + PA PB + kCTΛBTP + kΛC −2Λ

]

< 0

then globally asymptotically stable (any initial state x(0) convergesto the origin), where Λ = diag{λi} with λi ≥ 0.


Synchronization of Lur’e systems

• Master-slave synchronization scheme (drive-response):

M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + K(x − z)

Master system M drives slave system S (follows behaviour imposed bythe master system): under which conditions do the systems M and Ssynchronize?

(studies in synchronization of chaotic systems, and applications to secure

communications [Pecora & Carroll, 1990; Chen & Dong, 1998; Yalcin et al., 2005])

• Mutual synchronization scheme:

M1 : x = Ax + Bσ(Cx) + K1(x − z)M2 : z = Az + Bσ(Cz) + K2(x − z)

Systems M1 and M2 mutually influence each other.


Synchronization example

Master system

No synchronization Synchronization


Error system

• Consider the error e = x − z relative between the master M and theslave S system:

e = (A − K)e + B[σ(C(e + z)) − σ(Cz)]

• Assume a sector condition on σ(C(e + z)) − σ(Cz)[Suykens & Vandewalle, IJBC 1997; Curran, Suykens, Chua, IJBC 1997]

• A sufficient condition for global asymptotic stability of the error systemcan be obtained by taking e.g. a quadratic Lyapunov function

V (e) = eTPe, P = P T > 0

and derive under which condition dVdt

< 0, ∀e ∈ Rn0 .


Interpretation as a control problem

• Master-slave synchronization scheme:

M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + u

C : u = K(x − z)

with control signal u.

• Control objective: for given matrices A,B,C design a controller C withmatrix K such that synchronization is achieved.

• For Lur’e systems synchronization can be characterized by LMIs.

• Synchronization can be achieved for any choice of initial states x(0), z(0):for all initial state choices the systems synchronize in the sense that‖x(t) − z(t)‖ → 0 when time t → ∞.


Different control problems and approaches

• Dynamic measurement feedback control instead of full state feedback:if one cannot measure complete state vectors x, z.

• Robust synchronization: A, B,C matrices non-identical for master andslave system: it is possible to synchronize two systems up to a smallsynchronization error (e.g. limit cycle versus chaos); control in thepresence of disturbances or noise (e.g. H∞ control)

• Control via impulses (sporadic coupling, only from time to time andnon-equidistantly in time) instead of continuously controlling

• Control in systems with time-delays

• Other forms of synchronization: partial synchronization, clustersynchronization, phase synchronization, connection with graph topology

[Chen et al.; Wu et al.; Suykens et al.; Nijmeijer et al.; Yalcin et al.]


Problems in synchronization theory

IMPULSIVECOUPLING

Robust

Impulsive

Time-delaySynchronization

Synchronization

Synchronization

Synchronization

Synchronization

Nonlinear H∞

Robust Nonlinear H∞

EXTERNALINPUT

MISMATCHPARAMETER

AutonomousNon-autonomous

Design Purposes

Master-slave Synchronization Schemes

DELAY

Chaotic Lur’e Systems

[Yalcin et al., 2005]


Overview


• Cluster synchronization and community detection in complexnetworks

• Optimization using coupled local minimizers, cooperative behaviour


Complex networks

Random network Scale−free network

Number of links Number of linksNumber of links

Num

ber

of n

odes

Num

ber

of n

odes

[log scale]

[log

sca

le]

Num

ber

of n

odes

[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]

- Random networks: bell curve distribution- Scale-free networks: power law distribution

Robust against accidental failures, but vulnerable to coordinated attacks

Biological networks: growth (gene duplication) and preferential attachement

(rich-gets-richer mechanism: new nodes prefer to link to the more connected nodes)


Map of protein-protein interactions

[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]Highly linked proteins (network hubs) tend to be crucial for cell survival.

Only few proteins are able to physically attach to a huge number.

www.nd.edu/∼networks


Wave phenomena in neuronal networks

- Hodgkin-Huxley type model of oscillatory activity in the bursting neurons of a snail

- Burst waves of antiphase spiking excitation in a 200× 200 lattice of electrically coupled

nonidentical neurons (snapshots at different times)

[Komarov, Osipov, Suykens, Chaos 2008]


Synchronization in complex networks

• Synchronization of chaotic systems [Pecora & Carroll, 1990]:mainly low dimensional systems and regular network topologies

• Complex networks: larger networks, different network topologies

• Complex networks:- relation between network topology and synchronization into clusters?- how to design to achieve desired clusters?- how to cope with time delays or communication constraints?- how to enhance synchronizability of complex networks- how to rewire the network?- ...

[Suykens & Osipov, Focus issue, Chaos 2008; Arenas et al., PR 2008]


Link between synchronization and spectral clustering

• (generalized) Kuramoto model: N coupled phase oscillators

dθi

dt= ωi +

∑

j

Kij sin(θj − θi), i = 1, ..., N

Special case: ωi = ω, Kij = σaij with adjacency matrix [aij]

• Linearized dynamics (Laplacian matrix L)

dθi

dt= −σ

∑

j

Lijθj, i = 1, ..., N

• Relationship between topological scales and dynamic time scalesModular structures emerge at different time scales

[Arenas et al., PRL 2006, PR 2008]


Complex networks

Synchronization

Spectral clustering


Spectral clustering

SVM, kernel methods

Data


Complex networks

Synchronization

Spectral clustering

SVM, kernel methods

Data


Community detection from synchronization

• Kuramoto model: θi = ω + σ∑

j aij sin(θj − θi)

• Follow the evolution of

ρij(t) = 〈cos[θi(t) − θj(t)]〉

averaged over different initial conditions.

• Community detection based on a binary dynamic connectivity matrix

[Dt(T )]ij = 1 if ρij(t) > T, zero otherwise

T large enough: one finds set of disconnected clustersT smaller: inter-community connections become visible

• Other approach: matrix DT (t) unravels the topological structure of thenetwork at different time scales.

[Arenas et al., PRL 2006, PR 2008]


Finding communities in weighted networks (1)

0 0.2 0.4 0.60

0.5

1

1.5

2

2.5

3

time

Qw

0

0.5

1

1.5

2

2.5

3

Qw

=0.4947

Synthetic example [Lou & Suykens, Chaos 2011]: community detectionby considering [D]ij = tij if ρij(t) > T and zero otherwise, where tij isthe time needed for nodes i and j to synchronization in the sense thatρij(t) > T .



1030

24

16

19

23

21

15

9

1312

20

17

11

22

4

7

5

6

28 25 32

2926

14

8 3

2

1

18

31

3334

27

C3

C2C

1

C4

(a)

6 7 17 5 11 1 12 18 2 8 14 20 3 4 13 22 9 10 31 15 33 34 19 21 16 24 28 27 30 23 25 26 32 290

0.5

1

1.5

2

2.5

3

3.5

time

(b)

Qw

=0.4439

on the Zachary’s karate club network [Lou & Suykens, Chaos 2011]



on the American football team network [Lou & Suykens, Chaos 2011]


”Programming” clusters into complex networks

- cluster design on a 20 × 60 lattice of identical Rossler oscillators.- cluster ”CHAOS” obtained from randomly distributed initial conditions.

[Belykh, Osipov, Petrov, Suykens, Vandewalle, Chaos 2008]


Overview


• Cluster synchronization and community detection in complex networks

• Optimization using coupled local minimizers,cooperative behaviour


Optimization

Local optimization

+ fast

- local optimum

Newton, QN, LM, CG


Optimization

Local optimization

+ fast

- local optimum

Newton, QN, LM, CG

Global optimization

- slow

+ global search

GA, SA, swarms


Optimization

Local optimization

+ fast

- local optimum

Newton, QN, LM, CG

???

+ fast

+ global search

???

Global optimization

- slow

+ global search

GA, SA, swarms


Local optimization

• Consider the unconstrained optimization problem:

minx∈Rn

U(x)

with cost function U(·) continuously differentiable.

• Simple continuous-time steepest descent algorithm:

x = −η∇xU(x)

converging to a local optimum.

• Better local optimization methods:momentum term, Newton method, conjugate gradients, ...


Coupled local minimizers

• Essential idea for Coupled Local Minimizers (CLM):

1. consider two (or more) local optimizers and let them interact2. enforce that the optimizers should reach the same final state,

i.e. require state synchronization

• Realizing cooperative behaviour for optimization: based on coupling ofoptimization processes and master-slave synchronization

• Hierarchical scheme: objectives (cost functions) at the individual leveland at the group level

[Suykens et al., IJBC 2001]


Coupled local minimizers

weight space

cost

Multi−start local optimization

No interaction

weight space

cost

Coupled Local Minimizers

Interaction and information exchange



Array consisting of coupled local minimizers

space

space

cost

cost


CLM: a toy example

• Example: consider the following objective

minx,z

U(x) + U(z) subject to x = z

Lagrange programming network:

x = −∇xU(x) − (x − z) − λ

z = −∇zU(z) + (x − z) + λ

λ = x − z


Toy example: double potential well

−6 −4 −2 0 2 4 60

100

200

300

400

500

600

700

800

900

x

U(x

)

0 20 40 60 80 100 120 140 160 180 200−10

−5

0

5

10

15

20

25

30

t

x,z,λ

The initial states x(0), z(0) are chosen to be in the two different valleys.The states x(t), z(t) converge to the global solution at x = z = −2.9

(blue: x(t) - red: z(t) - green: λ(t))


Lagrange programming network

• Problem statement:

minx∈Rn

f(x) subject to h(x) = 0

• Lagrangian: L(x, λ) = f(x) + λTh(x)

• Lagrange programming network:

{

x = −∇xL(x, λ)

λ = ∇λL(x, λ)

This can be viewed as a continuous-time optimization algorithm.

[Zhang & Constantinides, 1992]


CLM: more general formulation (1)

• Consider a group consisting of q optimizers {x(i)}qi=1

• Minimize average energy cost subject to pairwise synchronization states

minx(i)∈Rn

1

q

q∑

i=1

U(x(i))

subject to x(i) − x

(i+1) = 0, i = 1, 2, ..., q

• Boundary conditions x(0) = x

(q), x(q+1) = x

(1)



CLM: more general formulation (2)

• Augmented Lagrangian (synchronization as hard and soft constraint)

L(x(i)

, λ(i)

) =η

q

qX

i=1

U [x(i)

] +1

2

qX

i=1

γi ‖x(i) − x

(i+1)‖22+

qX

i=1

〈λ(i), [x

(i) − x(i+1)

]〉

• Lagrange programming network:

{

x(i) = −η

q∇

x(i)U [x(i)] + γi−1[x

(i−1) − x(i)] − γi[x

(i) − x(i+1)] + λ(i−1) − λ(i)

λ(i) = x(i) − x

(i+1) , i = 1, 2, ..., q


Optimal cooperation

• Decrease of ensemble energy cost:

d〈U〉

dt=

1

q

qX

i=1

〈∂U [x(i)]

∂x(i)

, x(i)〉

=1

q

qX

i=1

〈∂U [x(i)]

∂x(i)

,−η

q

∂U [x(i)]

∂x(i)

+ γi−1[x(i−1) − x

(i)]

−γi[x(i) − x

(i+1)] + λ

(i−1) − λ(i)〉

• Optimal cooperation: LP problem in γ (scheduling of γi values)

minγ∈Rq

d〈U〉

dt|x,λ such that γ < γi < γ, i = 1, 2, ..., q

This incorporates the principle of master-slave synchronization.


Example: optimization of Lennard-Jones clusters

• In predicting 3D structure of proteins from amino acid sequences,potential energy surface (PES) minimization is often related to thenative structure of the protein.Benchmark problem: optimization of Lennard-Jones (LJ) clusters [Sali,1994; Wales, 1997, 1999].

• Cost function:

ULJ = 4∑

i<j

(1

r12ij

−1

r6ij

)

with rij the Euclidean distance between atom i and j (j = 1, ..., N).

• (LJ)38 which possesses a double-funnel energy landscape and is knownto be an interesting test-case [Wales, 1997, 1999].

• Important role of p(x(0)) ∝ exp[− 12σ2x(0)T

x(0)] (similar to consideringa confining potential in effective potential minimization methods).


Case (LJ)38

0 0.5 1 1.5 2 2.5 3

x 10−7

100

102

104

106

108

1010

1012

1014

t

ULJde

lta

Evolution of the cost function for q = 50 coupled local minimizers, reachingthe global minimum configuration for (LJ)38 with double-funnel landscape.


Case (LJ)150

Potential for application to larger scale problems


Example: CLM training of MLP neural networks

• CLM with state vectors x(i) (i = 1, ..., q) equal to the unknown weight

vectors θ(i) of the MLP.

• CLM training process corresponds to coupled backpropagation processeswith weight vector synchronization.

• The initial distribution of p(x(i)(0)) (i = 1, ..., q) (at time 0) plays animportant role, similar to the choice of a regularization constant (inmethods of minimizing errors and keeping the weights small).


CLM training of neural networks (1)

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

x

y

- MLP training (10 hidden units) of a sinusoidal function (green) given 20 noisy data

- Application of scaled CG without early stopping leading to overfitting (red) and best

result by Bayesian learning with regularization (blue).



−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

2

x

y

CLM result which optimizes a sum squared error on training data withoutregularization of the cost function.



0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

x 10−5

1

1.5

2

2.5

3

3.5

4

t

U

CLM evolution (group of q = 20 optimizers) of the sum squared error costfunction during optimization


Alternative Formulation to CLMs

• Capture a group of optimizers within a ball and shrink the ball

• Objective:

minx(i)∈Rn,r∈R

〈U〉 + 12 ν r2

subject to ‖x(i) − x(i+1)‖2

2 ≤ r2, i = 1, ..., q

where 〈U〉 = 1q

∑qi=1 U [x(i)].

• Advantage: always easy to find a feasible point to the constraints duringthe optimization process.

[Suykens & Vandewalle, 2002]


CLM: extensions

• Coupled Newton methods with applications in civil engineering[Teughels, De Roeck, Suykens, 2002]

• Additional noise can be injected into the system [Gunel et al., 2006]

• Extensions to coupled simulated annealing processes with cost functionevaluations only [Xavier-de-Souza, Suykens, Vandewalle, Bolle, IEEE-SMC-B 2010].

Successfully applied e.g. for tuning parameter selection in kernel methods,being more efficient than grid search, SA or GA [K. De Brabanter et al.,CSDA 2010].

• Stability analysis of CLMs [Lou & Suykens, IEEE-TCAS-I, in press].

• Hybrid CLMs: occasional impulsive coupling, suitable for parallelimplementations [Lou & Suykens, 2012].


Conclusions

• Synchronization phenomena: naturally happening in a wide range ofsystems and complex networks.

• Lur’e systems: broad class of nonlinear systems, conditions for globalstability and global synchronization can be obtained.

• Community detection in complex networks: obtainable also through asynchronization process

• Coupled local minimizers: aims for global search together with fasterconvergence.


Acknowledgements (1)

• Colleagues at ESAT-SCD (especially research units: systems, models,control - biomedical data processing - bioinformatics):

C. Alzate, A. Argyriou, J. De Brabanter, K. De Brabanter, L. De Lathauwer, B. De

Moor, M. Diehl, Ph. Dreesen, M. Espinoza, T. Falck, D. Geebelen, X. Huang, B.

Hunyadi, A. Installe, V. Jumutc, P. Karsmakers, R. Langone, J. Lopez, J. Luts, R.

Mall, S. Mehrkanoon, M. Moonen, Y. Moreau, K. Pelckmans, J. Puertas, L. Shi, M.

Signoretto, P. Tsiaflakis, V. Van Belle, R. Van de Plas, S. Van Huffel, J. Vandewalle,

T. van Waterschoot, C. Varon, S. Yu, and others

• L. Chua, P. Curran, A. Huang, T. Yang, A. Munuzuri, M. Yalcin, S.Gunel, S. Ozoguz, G. Osipov, M. Komarov, V. Belykh, V. Petrov, S.Xavier-de-Souza, X. Lou, A. Teughels, G. De Roeck, S. Arnout.

• Support from ERC AdG A-DATADRIVE-B, KU Leuven, GOA-MaNet,COE Optimization in Engineering OPTEC, IUAP DYSCO, FWO projects,IWT, IBBT eHealth, COST


Acknowledgements (2)


Thank you


Documents

Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour 18 Wave phenomena in neuronal networks - Hodgkin-Huxley type model of oscillatory