Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour...

Complex networks, synchronization and

cooperative behaviour

Johan Suykens

KU Leuven, ESAT-SCD/SISTAKasteelpark Arenberg 10

B-3001 Leuven (Heverlee), BelgiumEmail: johan.suykens@esat.kuleuven.be

http://www.esat.kuleuven.be/scd/

VUB Leerstoel 2012-2013 - Oct. 31 2012

Complex networks, synchronization and cooperative behaviour

Introduction

http://www.youtube.com”synchronization of metronomes”

(a modern version of the synchronization of two pendulum clocks observedby Christiaan Huygens, 1665)

Complex networks, synchronization and cooperative behaviour 1

Overview

• Chaotic systems synchronization, Lur’e systems

• Cluster synchronization and community detection in complex networks

• Optimization using coupled local minimizers, cooperative behaviour

Circuits and systems: Chua’s circuit

− −

vC2C2 C1

gNR(vC1

Chua’s circuit [Chua et al., 1986]: in dimensionless form

x = α(y − x − f(x))y = x − y + z

z = −βy

f(x) = m1x +1

2(m0 − m1) (|x + 1| − |x − 1|)

(depending on α, β: bistability, limit cycles, chaos)

Bifurcation to Chaos

(vC1, vC2)-plane:

Power spectrum vC1:

−→ birth of the double scroll attractor −→

Lur’e system

σ(·)

m(t) = 0

y(t)++

• Lur’e system:

x = Ax + Bu

y = Cx

u = σ(y)→ x = Ax + Bσ(Cx)

where x ∈ Rn and σ(·) : R

h → Rh satisfies a sector condition.

• Chua’s circuit: h = 1

More hidden units

• Multi-stability & Multi-scroll chaos:Extend the nonlinearity andcreate additional equilibrium points[Suykens & Vandewalle, 1991; Arena, 1996; Yalcin, 2001; Lu, 2006]

• Multilayer neural networks are universal approximators [Hornik, 1989]

(Chua’s circuit has 1 hidden unit (h = 1), more hidden units for multi-scrolls)

A gallery of multi-scroll attractors

[Suykens & Vandewalle, 1991; Yalcin et al., 2001]

Lur’e systems: examples

• Lur’e system:

x = Ax + Bu

y = Cx

u = σ(y)→ x = Ax + Bσ(Cx)

• Many examples of Lur’e systems in different areas:- Recurrent neural networks (Hopfield network: A = −I, C = I) [Hopfield, 1985]

- Cellular neural networks (sparse and structured matrices A, B, C) [Chua, 1988]

- Actuator saturation in control systems

- Chua’s circuit, multi-scroll circuits

- Arrays of coupled networks

- Genetic oscillator models

σ(·)

m(t) = 0

y(t)++

Genetic oscillators

A general genetic oscillator form [Li, Chen, Aihara, 2006]:

x(t) = Ax(t) +

Bifi(x(t))

• x(t) ∈ Rn: concentrations of proteins, RNAs, chemical complexes

• fi(x(t)) = [fi1(x1(t)); ...; fin(xn(t))]: regulatory function (monotonicallyincreasing or decreasing: e.g. Michaelis-Menten or Hill form)

Examples: Goodwin model, repressilator, toggle switch, circadian oscillators

Stability analysis and LMIs (1)

• Linear system:x = Ax

Quadratic Lyapunov function:

V = xTPx, P = P T > 0

• Stability analysis:

V = xTPx + xTP x = xT (ATP + PA)x < 0

Global asymptotic stability for

ATP + PA < 0

Linear matrix inequality (LMI) for a given matrix A [Boyd et al., 1994]

Stability analysis and LMIs (2)

• Lur’e system:x = Ax + Bσ(Cx)

Try e.g. a quadratic Lyapunov function (leading to a sufficient stabilitycondition):

V = xTPx, P = P T > 0

• Stability analysis: exploit the fact that σ belongs to sector [0, k]

V = xTPx + xTP x

≤ xTPx + xTP x−∑

i 2λiσi(σi − kcTi x) = [xTσT ]Z

ATP + PA PB + kCTΛBTP + kΛC −2Λ

then globally asymptotically stable (any initial state x(0) convergesto the origin), where Λ = diag{λi} with λi ≥ 0.

Synchronization of Lur’e systems

• Master-slave synchronization scheme (drive-response):

M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + K(x − z)

Master system M drives slave system S (follows behaviour imposed bythe master system): under which conditions do the systems M and Ssynchronize?

(studies in synchronization of chaotic systems, and applications to secure

communications [Pecora & Carroll, 1990; Chen & Dong, 1998; Yalcin et al., 2005])

• Mutual synchronization scheme:

M1 : x = Ax + Bσ(Cx) + K1(x − z)M2 : z = Az + Bσ(Cz) + K2(x − z)

Systems M1 and M2 mutually influence each other.

Synchronization example

Master system

No synchronization Synchronization

Error system

• Consider the error e = x − z relative between the master M and theslave S system:

e = (A − K)e + B[σ(C(e + z)) − σ(Cz)]

• Assume a sector condition on σ(C(e + z)) − σ(Cz)[Suykens & Vandewalle, IJBC 1997; Curran, Suykens, Chua, IJBC 1997]

• A sufficient condition for global asymptotic stability of the error systemcan be obtained by taking e.g. a quadratic Lyapunov function

V (e) = eTPe, P = P T > 0

and derive under which condition dVdt

< 0, ∀e ∈ Rn0 .

Interpretation as a control problem

• Master-slave synchronization scheme:

M : x = Ax + Bσ(Cx)S : z = Az + Bσ(Cz) + u

C : u = K(x − z)

with control signal u.

• Control objective: for given matrices A,B,C design a controller C withmatrix K such that synchronization is achieved.

• For Lur’e systems synchronization can be characterized by LMIs.

• Synchronization can be achieved for any choice of initial states x(0), z(0):for all initial state choices the systems synchronize in the sense that‖x(t) − z(t)‖ → 0 when time t → ∞.

Different control problems and approaches

• Dynamic measurement feedback control instead of full state feedback:if one cannot measure complete state vectors x, z.

• Robust synchronization: A, B,C matrices non-identical for master andslave system: it is possible to synchronize two systems up to a smallsynchronization error (e.g. limit cycle versus chaos); control in thepresence of disturbances or noise (e.g. H∞ control)

• Control via impulses (sporadic coupling, only from time to time andnon-equidistantly in time) instead of continuously controlling

• Control in systems with time-delays

• Other forms of synchronization: partial synchronization, clustersynchronization, phase synchronization, connection with graph topology

[Chen et al.; Wu et al.; Suykens et al.; Nijmeijer et al.; Yalcin et al.]

Problems in synchronization theory

IMPULSIVECOUPLING

Robust

Impulsive

Time-delaySynchronization

Synchronization

Nonlinear H∞

Robust Nonlinear H∞

EXTERNALINPUT

MISMATCHPARAMETER

AutonomousNon-autonomous

Design Purposes

Master-slave Synchronization Schemes

Chaotic Lur’e Systems

[Yalcin et al., 2005]

Overview

• Cluster synchronization and community detection in complexnetworks

• Optimization using coupled local minimizers, cooperative behaviour

Complex networks

Random network Scale−free network

Number of links Number of linksNumber of links

[log scale]

[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]

- Random networks: bell curve distribution- Scale-free networks: power law distribution

Robust against accidental failures, but vulnerable to coordinated attacks

Biological networks: growth (gene duplication) and preferential attachement

(rich-gets-richer mechanism: new nodes prefer to link to the more connected nodes)

Map of protein-protein interactions

[Barabasi & Bonabeau, 2003; Barabasi & Oltvai, 2004]Highly linked proteins (network hubs) tend to be crucial for cell survival.

Only few proteins are able to physically attach to a huge number.

www.nd.edu/∼networks

Wave phenomena in neuronal networks

- Hodgkin-Huxley type model of oscillatory activity in the bursting neurons of a snail

- Burst waves of antiphase spiking excitation in a 200× 200 lattice of electrically coupled

nonidentical neurons (snapshots at different times)

[Komarov, Osipov, Suykens, Chaos 2008]

Synchronization in complex networks

• Synchronization of chaotic systems [Pecora & Carroll, 1990]:mainly low dimensional systems and regular network topologies

• Complex networks: larger networks, different network topologies

• Complex networks:- relation between network topology and synchronization into clusters?- how to design to achieve desired clusters?- how to cope with time delays or communication constraints?- how to enhance synchronizability of complex networks- how to rewire the network?- ...

[Suykens & Osipov, Focus issue, Chaos 2008; Arenas et al., PR 2008]

Link between synchronization and spectral clustering

• (generalized) Kuramoto model: N coupled phase oscillators

dt= ωi +

Kij sin(θj − θi), i = 1, ..., N

Special case: ωi = ω, Kij = σaij with adjacency matrix [aij]

• Linearized dynamics (Laplacian matrix L)

dt= −σ

Lijθj, i = 1, ..., N

• Relationship between topological scales and dynamic time scalesModular structures emerge at different time scales

[Arenas et al., PRL 2006, PR 2008]

Complex networks

Synchronization

Spectral clustering

SVM, kernel methods

Complex networks

Synchronization

Spectral clustering

SVM, kernel methods

Community detection from synchronization

• Kuramoto model: θi = ω + σ∑

j aij sin(θj − θi)

• Follow the evolution of

ρij(t) = 〈cos[θi(t) − θj(t)]〉

averaged over different initial conditions.

• Community detection based on a binary dynamic connectivity matrix

[Dt(T )]ij = 1 if ρij(t) > T, zero otherwise

T large enough: one finds set of disconnected clustersT smaller: inter-community connections become visible

• Other approach: matrix DT (t) unravels the topological structure of thenetwork at different time scales.

[Arenas et al., PRL 2006, PR 2008]

Finding communities in weighted networks (1)

0 0.2 0.4 0.60

=0.4947

Synthetic example [Lou & Suykens, Chaos 2011]: community detectionby considering [D]ij = tij if ρij(t) > T and zero otherwise, where tij isthe time needed for nodes i and j to synchronization in the sense thatρij(t) > T .

28 25 32

6 7 17 5 11 1 12 18 2 8 14 20 3 4 13 22 9 10 31 15 33 34 19 21 16 24 28 27 30 23 25 26 32 290

=0.4439

on the Zachary’s karate club network [Lou & Suykens, Chaos 2011]

on the American football team network [Lou & Suykens, Chaos 2011]

”Programming” clusters into complex networks

- cluster design on a 20 × 60 lattice of identical Rossler oscillators.- cluster ”CHAOS” obtained from randomly distributed initial conditions.

[Belykh, Osipov, Petrov, Suykens, Vandewalle, Chaos 2008]

Overview

• Cluster synchronization and community detection in complex networks

• Optimization using coupled local minimizers,cooperative behaviour

Optimization

Local optimization

+ fast

- local optimum

Newton, QN, LM, CG

Optimization

Local optimization

+ fast

- local optimum

Newton, QN, LM, CG

Global optimization

- slow

+ global search

GA, SA, swarms

Optimization

Local optimization

+ fast

- local optimum

Newton, QN, LM, CG

+ fast

+ global search

Global optimization

- slow

+ global search

GA, SA, swarms

Local optimization

• Consider the unconstrained optimization problem:

minx∈Rn

with cost function U(·) continuously differentiable.

• Simple continuous-time steepest descent algorithm:

x = −η∇xU(x)

converging to a local optimum.

• Better local optimization methods:momentum term, Newton method, conjugate gradients, ...

Coupled local minimizers

• Essential idea for Coupled Local Minimizers (CLM):

1. consider two (or more) local optimizers and let them interact2. enforce that the optimizers should reach the same final state,

i.e. require state synchronization

• Realizing cooperative behaviour for optimization: based on coupling ofoptimization processes and master-slave synchronization

• Hierarchical scheme: objectives (cost functions) at the individual leveland at the group level

[Suykens et al., IJBC 2001]

Coupled local minimizers

weight space

Multi−start local optimization

No interaction

weight space

Coupled Local Minimizers

Interaction and information exchange

Array consisting of coupled local minimizers

CLM: a toy example

• Example: consider the following objective

minx,z

U(x) + U(z) subject to x = z

Lagrange programming network:

x = −∇xU(x) − (x − z) − λ

z = −∇zU(z) + (x − z) + λ

λ = x − z

Toy example: double potential well

−6 −4 −2 0 2 4 60

0 20 40 60 80 100 120 140 160 180 200−10

x,z,λ

The initial states x(0), z(0) are chosen to be in the two different valleys.The states x(t), z(t) converge to the global solution at x = z = −2.9

(blue: x(t) - red: z(t) - green: λ(t))

Lagrange programming network

• Problem statement:

minx∈Rn

f(x) subject to h(x) = 0

• Lagrangian: L(x, λ) = f(x) + λTh(x)

• Lagrange programming network:

x = −∇xL(x, λ)

λ = ∇λL(x, λ)

This can be viewed as a continuous-time optimization algorithm.

[Zhang & Constantinides, 1992]

CLM: more general formulation (1)

• Consider a group consisting of q optimizers {x(i)}qi=1

• Minimize average energy cost subject to pairwise synchronization states

minx(i)∈Rn

U(x(i))

subject to x(i) − x

(i+1) = 0, i = 1, 2, ..., q

• Boundary conditions x(0) = x

(q), x(q+1) = x

CLM: more general formulation (2)

• Augmented Lagrangian (synchronization as hard and soft constraint)

L(x(i)

, λ(i)

U [x(i)

γi ‖x(i) − x

(i+1)‖22+

〈λ(i), [x

(i) − x(i+1)

• Lagrange programming network:

x(i) = −η

x(i)U [x(i)] + γi−1[x

(i−1) − x(i)] − γi[x

(i) − x(i+1)] + λ(i−1) − λ(i)

λ(i) = x(i) − x

(i+1) , i = 1, 2, ..., q

Optimal cooperation

• Decrease of ensemble energy cost:

d〈U〉

〈∂U [x(i)]

∂x(i)

, x(i)〉

〈∂U [x(i)]

∂x(i)

,−η

∂U [x(i)]

∂x(i)

+ γi−1[x(i−1) − x

−γi[x(i) − x

(i+1)] + λ

(i−1) − λ(i)〉

• Optimal cooperation: LP problem in γ (scheduling of γi values)

minγ∈Rq

d〈U〉

dt|x,λ such that γ < γi < γ, i = 1, 2, ..., q

This incorporates the principle of master-slave synchronization.

Example: optimization of Lennard-Jones clusters

• In predicting 3D structure of proteins from amino acid sequences,potential energy surface (PES) minimization is often related to thenative structure of the protein.Benchmark problem: optimization of Lennard-Jones (LJ) clusters [Sali,1994; Wales, 1997, 1999].

• Cost function:

ULJ = 4∑

with rij the Euclidean distance between atom i and j (j = 1, ..., N).

• (LJ)38 which possesses a double-funnel energy landscape and is knownto be an interesting test-case [Wales, 1997, 1999].

• Important role of p(x(0)) ∝ exp[− 12σ2x(0)T

x(0)] (similar to consideringa confining potential in effective potential minimization methods).

Case (LJ)38

0 0.5 1 1.5 2 2.5 3

x 10−7

Evolution of the cost function for q = 50 coupled local minimizers, reachingthe global minimum configuration for (LJ)38 with double-funnel landscape.

Case (LJ)150

Potential for application to larger scale problems

Example: CLM training of MLP neural networks

• CLM with state vectors x(i) (i = 1, ..., q) equal to the unknown weight

vectors θ(i) of the MLP.

• CLM training process corresponds to coupled backpropagation processeswith weight vector synchronization.

• The initial distribution of p(x(i)(0)) (i = 1, ..., q) (at time 0) plays animportant role, similar to the choice of a regularization constant (inmethods of minimizing errors and keeping the weights small).

CLM training of neural networks (1)

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−0.5

- MLP training (10 hidden units) of a sinusoidal function (green) given 20 noisy data

- Application of scaled CG without early stopping leading to overfitting (red) and best

result by Bayesian learning with regularization (blue).

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−0.5

CLM result which optimizes a sum squared error on training data withoutregularization of the cost function.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

x 10−5

CLM evolution (group of q = 20 optimizers) of the sum squared error costfunction during optimization

Alternative Formulation to CLMs

• Capture a group of optimizers within a ball and shrink the ball

• Objective:

minx(i)∈Rn,r∈R

〈U〉 + 12 ν r2

subject to ‖x(i) − x(i+1)‖2

2 ≤ r2, i = 1, ..., q

where 〈U〉 = 1q

∑qi=1 U [x(i)].

• Advantage: always easy to find a feasible point to the constraints duringthe optimization process.

[Suykens & Vandewalle, 2002]

CLM: extensions

• Coupled Newton methods with applications in civil engineering[Teughels, De Roeck, Suykens, 2002]

• Additional noise can be injected into the system [Gunel et al., 2006]

• Extensions to coupled simulated annealing processes with cost functionevaluations only [Xavier-de-Souza, Suykens, Vandewalle, Bolle, IEEE-SMC-B 2010].

Successfully applied e.g. for tuning parameter selection in kernel methods,being more efficient than grid search, SA or GA [K. De Brabanter et al.,CSDA 2010].

• Stability analysis of CLMs [Lou & Suykens, IEEE-TCAS-I, in press].

• Hybrid CLMs: occasional impulsive coupling, suitable for parallelimplementations [Lou & Suykens, 2012].

Conclusions

• Synchronization phenomena: naturally happening in a wide range ofsystems and complex networks.

• Lur’e systems: broad class of nonlinear systems, conditions for globalstability and global synchronization can be obtained.

• Community detection in complex networks: obtainable also through asynchronization process

• Coupled local minimizers: aims for global search together with fasterconvergence.

Acknowledgements (1)

• Colleagues at ESAT-SCD (especially research units: systems, models,control - biomedical data processing - bioinformatics):

C. Alzate, A. Argyriou, J. De Brabanter, K. De Brabanter, L. De Lathauwer, B. De

Moor, M. Diehl, Ph. Dreesen, M. Espinoza, T. Falck, D. Geebelen, X. Huang, B.

Hunyadi, A. Installe, V. Jumutc, P. Karsmakers, R. Langone, J. Lopez, J. Luts, R.

Mall, S. Mehrkanoon, M. Moonen, Y. Moreau, K. Pelckmans, J. Puertas, L. Shi, M.

Signoretto, P. Tsiaflakis, V. Van Belle, R. Van de Plas, S. Van Huffel, J. Vandewalle,

T. van Waterschoot, C. Varon, S. Yu, and others

• L. Chua, P. Curran, A. Huang, T. Yang, A. Munuzuri, M. Yalcin, S.Gunel, S. Ozoguz, G. Osipov, M. Komarov, V. Belykh, V. Petrov, S.Xavier-de-Souza, X. Lou, A. Teughels, G. De Roeck, S. Arnout.

• Support from ERC AdG A-DATADRIVE-B, KU Leuven, GOA-MaNet,COE Optimization in Engineering OPTEC, IUAP DYSCO, FWO projects,IWT, IBBT eHealth, COST

Acknowledgements (2)

Thank you

Complex networks, synchronization and …Complex networks, synchronization and cooperative behaviour...

Documents

Pinning synchronization of delayed neural networks

Synchronization and Chimera States in Complex Networks · Yuri Maistrenko Academy of Sciences of Ukraine, Kiev, Ukraine E-mail: y.maistrenko@biomed.kiev.ua Lecture 14 Synchronization

Kuramoto Model of Synchronization in Complex Multilayer ...reu.dimacs.rutgers.edu/~kaylac/Cummings.Presentation1.Handout.pdf · Kuramoto Model of Synchronization in Complex Multilayer

Exploring Synchronization in Complex Oscillator Networkspeople.ee.ethz.ch/~floriand/docs/Slides/Tutorial_CDC... · 2020-06-10 · Exploring Synchronization in Complex Oscillator Networks

Exploring Synchronization in Complex Oscillator Networksmotion.me.ucsb.edu/pdf/2012i-db.pdf · 2013-08-28 · Exploring Synchronization in Complex Oscillator Networks Florian Dorﬂer

Algorithms for Sensor Localization and Synchronization in ... · time synchronization in wireless sensor networks. 2.1 Wireless Sensor Networks Wireless Sensor Networks are becoming

Gradient Clock Synchronization in Wireless Sensor Networks

Time Synchronization for Wireless Sensor Networks

COMMUNICATION AND SYNCHRONIZATION IN DISCONNECTED NETWORKS

Distributed Synchronization in Wireless Networks A · DISTRIBUTED SYNCHRONIZATION IN WIRELESS NETWORKS In the area of wireless networks, cellular telephony has monopo-lized the attention

Clock Synchronization in Sensor Networks

Symmetries, Clusters, and Synchronization Patterns in ... · Symmetries, Clusters, and Synchronization Patterns in Complex Networks Thomas E. Murphy Dept. of Electrical & Computer

Synchronization in complex oscillator networks and smart grids · Synchronization in complex oscillator networks and smart grids Florian Dörﬂera,b,1, Michael Chertkovb, and Francesco

Complex Networks using static routes Complex Networks Using Static Routes Documents... · 2014-02-20 · Complex Networks using static routes Complex Networks ... lThe command for

Time synchronization in sensor networks

NanJing University of Posts & Telecommunications Synchronization and Fault Diagnosis of Complex Dynamical Networks Guo-Ping Jiang College of Automation,

NetEvo Dynamical Complex Networks - Amazon S3...Evolving enhanced topologies for the synchronization of dynamical complex networks. PRE 81, 056212 (2010) [4] T.E. Gorochowski, et al

Stability of Synchronized Motion in Complex Networkstpereir1/LectureNotes.pdf · The analysis of synchronization in complex networks has beneﬁted from ad-vances in the understanding

Symmetries, Clusters, and Synchronization Patterns in Complex Networks

Synchronization of oscillatory networks in terms of global ... · Synchronization of oscillatory networks ... Synchronization of large ensembles of oscillators is an omnipresent phenomenon