Machine learning algorithms for predicting the amplitude ... · Machine learning algorithms for predicting the amplitude of chaotic laser pulses Machine learning algorithms for predicting

Machine learning algorithms for predicting the amplitude of chaotic laser pulses

Machine learning algorithms for predicting the amplitude ofchaotic laser pulses

Pablo Amil,1, a) Miguel C. Soriano,2 and Cristina Masoller11)Departament de Física, Universitat Politécnica de Catalunya, St. Nebridi 22, Terrassa 08222, Barcelona,Spain.2)Instituto de Física Interdisciplinar y Sistemas Complejos, IFISC (CSIC-UIB), Campus Universitat de les Illes Balears,E-07122 Palma de Mallorca, Spain.

(Dated: 21 October 2019)

Forecasting the dynamics of chaotic systems from the analysis of their output signals is a challenging problem withapplications in most fields of modern science. In this work, we use a laser model to compare the performance ofseveral machine learning algorithms for forecasting the amplitude of upcoming emitted chaotic pulses. We simulate thedynamics of an optically injected semiconductor laser that presents a rich variety of dynamical regimes when changingthe parameters. We focus on a particular dynamical regime that can show ultra-high intensity pulses, reminiscent ofrogue waves. We compare the goodness of the forecast for several popular methods in machine learning, namelydeep learning, support vector machine, nearest neighbors and reservoir computing. Finally, we analyze how theirperformance for predicting the height of the next optical pulse depends on the amount of noise and the length of thetime-series used for training.

Predicting the dynamical evolution of chaotic systems isan extremely challenging problem with important practi-cal applications. With unprecedented advances in com-puter science and artificial intelligence, many algorithmsare nowadays available for time series forecasting. Here,we use a well-known chaotic system of an optically in-jected semiconductor laser that exhibits fast and irregu-lar pulsing dynamics to compare the performance of sev-eral algorithms (deep learning, support vector machine,nearest neighbors and reservoir computing) for predictingthe amplitude of the next pulse. We compare the predic-tive power of such machine learning methods in terms ofdata requirements and the robustness towards the pres-ence of noise in the evolution of the system. Our resultsindicate that an accurate prediction of the amplitude ofupcoming chaotic pulses is possible using machine learn-ing techniques, although the presence of extreme events inthe time series and the consideration of stochastic contri-butions in the laser model bound the accuracy that can beachieved.

I. INTRODUCTION

Optically injected semiconductor lasers have a rich vari-ety of dynamical regimes, including stable locked emission,regular pulsing and chaotic behavior1,2. These regimes havefound several practical applications. For example, under sta-ble emission the laser emits light at the injected wavelength(the so-called injection-locking region) and has a high reso-nance frequency and a large modulation bandwidth3, whichhave broad applications for optical communications. The reg-ular pulsing regime can be used for microwave generation4,5,

a)Electronic mail: [email protected]

while the broad-band chaotic signal can be exploited for ultra-fast random number generation6.

In turn, the output of the laser in the chaotic regime can beused for testing new methods for data analysis, and in partic-ular, for time series prediction. Predicting the dynamical evo-lution of complex systems from the analysis of their outputsignals is an important problem in nonlinear science7–9, witha wide range of interdisciplinary applications. In these “bigdata” days, a significant number of researchers are focusingon developing novel methods for time series forecasting basedon machine learning algorithms10–14.

Delay embedding and recurrent neural networks have beenused to predict the evolution of chaotic systems such as theLorenz system and the Mackey-Glass system15. Locally lin-ear neurofuzzy models16 and support vector machine17 havealso been used to forecast chaotic signals. Here, in contrastwith previous works, we do not attempt to forecast the evolu-tion of a chaotic system, but the amplitude of the next peak inthe observed signal.

As a case study, we consider the dynamics of an opticallyinjected laser. We simulate the laser dynamics using a well-known rate equation model1,18, and use the chaotic regime tocompare the performance of several machine learning algo-rithms (deep learning, support vector machine, nearest neigh-bors and reservoir computing) for forecasting the amplitudeof the next intensity pulse.

Our main motivation to study this system is that it can beimplemented experimentally and we hope that our work willmotivate the analysis of real data. An important characteristicof this laser system is that it has control parameters (that canbe varied in the experiment) that allow to generate time serieswith or without extreme pulses.

Therefore, in the simulations, within the chaotic regime, weconsider two different situations: the intensity pulses displayoccasional extreme values (so-called optical rogue waves19,20)or the intensity pulses are irregular but do not display extremefluctuations. In the first case, the probability distribution func-tion (pdf) of pulse amplitudes is long tailed, while in the sec-

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

Machine learning algorithms for predicting the amplitude of chaotic laser pulses 2

ond case, it has a well-defined cut off.The possibility of predicting and suppressing extreme

pulses in a chaotic system has been demonstrated in21, but inthis work the authors did not attempt to predict the pulse am-plitude but rather the occurrence of a very high pulse when-ever the trajectory approached a particular region of the phasespace. To shed light on the limits of the forecast of extremeevents, we consider dynamical regimes with and without ex-treme pulses, produced by the same underlying system, andwe attempt to predict the amplitude of the next pulse, regard-less of whether it is normal or extreme. In our system wefind that, while both regular and extreme pulses can be fore-casted, the existence of extreme pulses bounds the predictionaccuracy. In an experimental setup, observational noise andthe limited bandwidth of the detection system (photodiode,oscilloscope) can further limit the predictability of the pulseamplitude.

II. MODEL

We simulated the dynamics of the complex optical field Eand the carrier population N in a semiconductor laser withoptical injection using the following rate equations2,22.

dEdt

= κ (1+ iα)(N−1)E +

+i∆ωE +√

Pin j +√

Dξ (t) , (1)dNdt

= γN

(µ−N−N |E|2

). (2)

The parameters in Eqs. 1-2 are: κ , the field decay rate,which we set at 300 ns−1; α , the linewidth enhancement fac-tor, which we set at 3; ∆ω , the optical frequency detuning,which we set at 2π × 0.49 GHz; Pin j, the optical injectionstrength, which we set to 60 ns−2; D, the noise level, which wevaried; γN , the carrier decay rate, which we set at 1 ns−1, µ thepump current parameter, which we varied. ξ (t) is a complexuncorrelated Gaussian noise of zero mean and unity variancethat represents spontaneous emission: ξ (t) = ξr(t) + iξi(t)with 〈ξr(t)ξr(t ′)〉 = δ (t − t ′), 〈ξi(t)ξi(t ′)〉 = δ (t − t ′) and〈ξr(t)ξi(t ′)〉= 0.

To simulate the evolution of Eqs. 1 and 2, we used theRunge-Kutta method of order 2 with a time step of 10−3 ns,as described in23, which takes into account the stochastic evo-lution with white noise. In this work, we will analyze thechaotic pulses that appear at the output intensity of the laserdefined as P = |E|2.

Figure 1 displays how the intensity deterministic dynamics(D = 0) depends on the pump current parameter µ . For smallµ (not shown), the laser emits a constant intensity, but as µ

increases a Hopf bifurcation and a series of period-doublingbifurcations occur, resulting in chaotic emission. Around µ =2.2, the intensity shows extreme pulses as shown in Fig. 1and the time series in Fig. 2(a and b). In contrast, at aroundµ = 2.45, the amplitude of the pulses in this chaotic regimeis tightly bounded as it can be seen in Fig. 1 and in the timeseries in Fig. 2(c and d).

FIG. 1. Deterministic bifurcation diagram of the output intensity ofthe injected laser (D=0) when varying the pump current parameterµ . For the subsequent analysis, we choose two currents that lead tovery different dynamical behaviours: (i) µ = 2.2, where the systempresents extreme events, and (ii) µ = 2.45 where the systems dis-plays a bounded chaotic behaviour. Example time series for theseparameters, including noise in the simulations (D = 10−4 ns−1) areshown in Fig. 2.

0 1 2 3 4 5 6 7 8

Time lag (peaks)

-0.5

0

0.5

1

-0.5

0

0.5

1

-0.5

0

0.5

1

-0.5

0

0.5

1

Autocorrelation

0 2 4 6 8 10 12 14 16 18 20

Time (ns)

0

2

0

2

0

2

4

6

8

0

2

4

Intensity

(A.U

.)

abcd

FIG. 2. (Left) Intensity time-series of the laser with optical injectionand (right) autocorrelation function of the extracted peak series forµ = 2.2 (a and b) and µ = 2.45 (c and d) and noise level of D = 0 (aand c) and D = 10−4 ns−1 (b and d).

The autocorrelation functions of the peak intensity values(i.e. the autocorrelation of the series yi built with the ampli-tude of each intensity peak) for both values of µ and bothvalues of D are shown in Fig. 2. For µ = 2.2, the autocorre-lation of the peak series decays to zero after a few peaks, bothfor D = 0 and D = 10−4 ns−1. It can be seen that for µ = 2.45,the autocorrelation of the peak series does not decay to zeroand shows non-negligible values of the autocorrelation evenafter 8 peaks. The values of the autocorrelation as a functionof the time lag (number of peaks) are larger for the series withnoise. We show in Fig. 2(d) that the evolution of the laser in-

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755


tensity with noise alternates regions of more regular behaviourwith regions of chaotic dynamics, which is not seen in the timeseries without noise in Fig. 2(c). This is due to the fact thatµ = 2.45 lies in a small chaotic island near regular regimes(see Fig. 1) and we find noise-induced jumps between dif-ferent dynamical regimes. We anticipate that the faster decayof the autocorrelation function, together with the presence ofextreme pulses, in the time series for µ = 2.2 will result onlarger prediction errors than in the time series for µ = 2.45.

III. FORECAST METHODS

All machine learning methods used here tackle the problemof function approximation. We use them to forecast the ampli-tude of the upcoming intensity peaks by assuming that there isan objective function (that we try to infer) that takes as inputsa certain number of consecutive peak amplitudes and returnsas output the amplitude of the next peak.

Except for the method of reservoir computing (that has aninternal state with memory of the history of the inputs), allother methods are memoryless (i.e. they have no internal stateof the history of the inputs), and explicit input and outputsof the objective function have to be provided in the trainingphase, providing information of the history with the previousintensity peaks amplitude. Let yi be the i-th intensity peakamplitude, our objective function is

f (yi−n, ...,yi−1) = yi, (3)

where n is the number of input intensity peak amplitudes thatthe machine learning algorithm is fed with. For the forecastof the peak amplitudes, we found that keeping n = 3 yieldedthe minimum prediction error and further increasing n pro-duced no accuracy enhancement. This choice will be justifiedin more detail in the results section (see Fig. 8).

For simplicity we call xi = (yi−n, ...,yi−1) and thus, we canrewrite eq. 3 as:

f (xi) = yi. (4)

For testing the methods, we use a different realization ofthe same simulations (not used in the training phase), and withthis new data we evaluated the learned function,

f (xi) = yi. (5)

Several statistical measures have been used in the literature toquantify the performance of time series prediction algorithmssuch as the correlation coefficient (CC)24, the mean squarederror (MSE)15, the normalized mean squared error (NMSE)16,the root mean squared error (RMSE)15, the normalized root-mean-square error (NRMSE)17 , the mean absolute relativeerror (MARE), etc. Here we use the MARE24 defined as:

MARE =1N

i=N

∑i=1

|yi− yi|yi

. (6)

In the following subsections we describe the different algo-rithms used.

A. Statistical methods

1. k-Nearest Neighbours

The k-Nearest Neighbours (KNN) is a popular method usedfor supervised learning25. It works by finding, in the trainingset, the k most similar points to a test point. Then, the predic-tion of the test point is obtained by averaging the response ofsuch k points (in the training set). Thus,

y =1k ∑

j∈N

y j, (7)

where N (the neighbourhood of the test point xi) is the set ofindexes of the k points in the training set that are closest to thetest point.

2. Support Vector Machine

Support Vector Machine26–28 (SVM), is another popularmethod used for supervised learning, which is based on theinner product of points in the set to approximate the re-sponse function29. Nonlinearities can be introduced straight-forwardly by modifying the inner product function. For linearSVM the inner product of two points (xi and x j) is calculatedas

⟨xi,x j

⟩= xt

ix j, (8)

while nonlinearity can be introduced by using a Gaussian ker-nel to calculate the inner product,

⟨xi,x j

⟩= exp

(−∥∥xi−x j

∥∥2σ2

). (9)

The objective function, f (xi) = yi, is written as a linearcombination of the inner products with the support vectors

f (xi) = ∑j

β j⟨x j,xi

⟩+b. (10)

The coefficients β j and b are obtained by solving a convexoptimization problem27.

The linear SVM has the advantage of being parameter-free.In contrast, for using the Gaussian kernel the scale factor σ

has to be defined. To set the value of σ we used the automaticheuristic implemented in the Statistics and Machine LearningToolbox of Matlab (the fitrsvm function).

B. Artificial neural networks

1. Feed-forward neural networks

Feed-forward neural networks, usually simply referred asneural networks, use a set of units, called perceptrons, that,

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755


FIG. 3. Graphical representation of a feed-forward neural networkwith a single hidden layer (a) and a reservoir computer (b). Thedashed lines represent the connections that can be adjusted via alearning procedure while the solid lines account for the connectionsthat are randomly weighted and left untrained.

when used in a large network, their output can approximatea great variety of functions depending on the weights of theconnections among the units.

Perceptrons perform two tasks, they compute a weightedsum of all their inputs (and a constant bias input), and theyperform a nonlinear function, called activation function, to theresult. The output of the activation function is the output ofthe perceptron. Most commonly, the activation functions usedare sigmoids, in this work we use the tanh function in all butthe last (output) layer, in which we don’t use a nonlinearity toavoid bounding the final output to the codomain of the non-linearity.

A feed-forward neural network, is a network of such per-ceptrons wherein they are ordered in layers, as shown in Fig.3(a) for a single hidden layer. The perceptrons of the first layerhave their inputs set to be the inputs of the whole network. Forthe rest of the layers, the inputs are defined as the outputs ofthe perceptrons in the previous layer.

The parameters of these networks are the weights of eachperceptron. These parameters can be set using a gradientdescend algorithm, in feed-forward neural networks an ef-ficient algorithm to perform gradient descend, called back-propagation30, may be used.

We used a shallow neural network (shallow NN), consistingof a single hidden layer of 30 perceptrons and a deep neuralnetwork (deep NN) consisting of 5 hidden layers of 10, 20,50, 25, and 10 perceptrons (ordered from the input layer tothe output layer), respectively.

2. Reservoir computing

Reservoir computing (RC) is a computational paradigm thatcan be viewed as a particular type of artificial neural networkswith a single hidden layer and recurrent connections31. Aring topology in the hidden layer (or reservoir), as the oneshown in Fig. 3(b), is a simple way to create recurrent connec-tions. Such a ring topology yields a performance comparable

to more complex network topologies in the reservoir32. Be-ing a recurrent neural network, the reservoir computing tech-nique is suitable to process sequential information. In reser-voir computing, the connection weights from the input layerto the hidden layer as well as the connection weights withinthe reservoir are drawn from a Gaussian distribution and leftuntrained. The connection weights from the reservoir to theoutput layer are trained in a supervised learning procedure,which translates to a linear problem that can be solved via asimple linear regression33.

The nodes in the reservoir layer perform a nonlineartransformation of the input data. Here we use a sinesquared nonlinearity, which can be implemented in photonichardware34,35, but other types of nonlinearity are also possi-ble. Finally, the output node performs a weighted sum of thereservoir outputs.

The RC method can be described by the following equa-tions for the states of the nodes in the hidden layer (z j) andthe prediction of the output node (y):

z ji = F(γwI

jyi−1 +β z j−1i−1 ), (11)

yi =D

∑j=1

wOj z j

i ; (12)

where i refers to the peaks in the laser time series, j is theindex of the node in the hidden layer, wI are the set of in-put weights drawn from a random Gaussian distribution, γ

and β are the input and feedback scaling, respectively, andF(u) = sin2(u + φ) is the nonlinear activation function. Ineq. 12, D is the number of hidden nodes and wO stands forthe trained output weights. In order to create a recurrent ringconnectivity in the hidden layer (also known as reservoir), weconnect node z j ( j = {2...D}) with its neighbour z j−1 and weclose the ring by connecting node z1 with zD as shown in Fig.3(b). Here, we have set the hyper-parameter values as γ = 4.5,β = 0.25, φ = 0.6π and D = 6000, which minimize the pre-diction error. We have verified that a tanh activation functionyields quantitatively similar results once the hyper-parametersγ and β are optimized.

We note that the heuristic for the RC practitioners is toassume a random interconnection topology in the reservoir,which usually yields good results. However, regular networktopologies also yield optimal results as long as the hyper-parameters are optimized36,37, as it has been the case here.For the RC method, we only feed a single amplitude valueto predict the amplitude of the next pulse. Feeding the RCmethod with the value of several previous peaks would meanthat, in practice, the reservoir computer would not need to useits own internal memory. The motivation to employ a differ-ent number of input peaks for the reservoir computer lies onthe observation that it can reach a prediction error compara-ble to the other methods without using explicit memory of thepreceding peaks.

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755


IV. RESULTS AND DISCUSSION

We now proceed to evaluate the performance of the dif-ferent forecast methods on the prediction of the amplitude ofchaotic laser pulses. The goal of our work is to predict the am-plitude of the upcoming laser pulse given the recent history ofthe dynamics. To that end, we generate long time series ofa laser subject to optical injection following the model de-scribed in Eqs. (1) and (2) for the two chaotic regimes shownin Fig. 2.

0 2 4 6

time (ns)

0

2

4

6

Intensity

Intensity signalLinear SVMGaussian SVMShallow NNDeep NNKNNRC

FIG. 4. Simulated intensity time series together with the peak am-plitudes predicted by the different methods. The parameters areD = 10−4 ns−1 and µ = 2.2. All methods were trained using 15000ns of simulation, which contain 65534 peak intensity values.

By looking at Fig. 2, the presence of extreme events in thetime series of the laser when the current is µ = 2.2 becomesapparent. We anticipate that the existence of such extremeevents poses a challenge for the prediction of the chaotic laserpulses’ amplitude. Fig. 4 shows a segment of the time seriesof the laser for the parameters µ = 2.2 and D = 10−4 ns−1

together with the prediction of the pulses amplitude for all themethods considered in this work. From this first qualitativeevaluation of the forecast methods, we can observe how thelinear SVM method is outperformed by the other methods.In turn, the methods Deep NN, KNN and RC tend to yielda similar, accurate, prediction of the amplitude of the chaoticpulses.

A further visualization of the goodness of the differentmethods is provided by the scatter plots in Fig. 5. These scat-ter plots represent the predicted peak intensities versus the realones. The methods with a better prediction accuracy need toalign to a diagonal line in this representation. For this chaoticregime of the laser dynamics with the presence of extremeevents, the Deep NN, KNN and RC methods are well alignedto the diagonal lines as shown in Figs. 5(d)-(f). In contrast,the Shallow NN and Gaussian SVM methods tend to under-estimate the amplitude of medium to large pulses as it canbe seen in 5(b)-(c). As shown in Fig. 5(a), the linear SVMmethod fails to capture the complexity of the dynamics.

When doing data-driven forecasting, it is necessary to eval-uate the number of training points needed to have accurate

FIG. 5. Scatter plots displaying the simulated peak intensity vs. thepredicted peak intensity for the methods (a) Linear SVM, (b) Gaus-sian SVM, (c) Shallow Neural Network, (d) Deep Neural Network,(e) k-Nearest Neighbors, (f) Reservoir Computing. The parametersare D = 10−4 ns−1 and µ = 2.2. All methods were trained using15000 ns of simulation, containing 65534 peak intensity values.

results. In Figs. 6 and 7 we show how the accuracy (as mea-sured by the mean absolute relative error, Eq. 6) depends onthe number of points used to train the algorithms, when thereare extreme pulses (Fig. 6) and when there are no extremepulses (Fig. 7).

First, we compare the forecast results for the noise-free nu-merical simulations at currents µ = 2.2 and µ = 2.45, whichare shown in Figs. 6(a) and 7(a). The MARE of the forecastfor µ = 2.2 is at least two orders of magnitude worse than theforecast for µ = 2.45. This is due to the added complexity ofthe extreme events at µ = 2.2, deteriorating the performanceof all the forecasting methods. We find that the KNN, DeepNN, and RC methods, in this order, yield the most accuratepredictions for µ = 2.2. These methods, together with theShallow NN, yield the lowest MARE for µ = 2.45. In bothcases, the performance of the RC method becomes more ac-curate when the number of training data points is larger than

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755


102 103 104 105 106

Training data points

10−4

10−3

10−2

10−1

100

MARE

(a)

Linear SVMGaussian SVMShallow NNDeep NNKNNRC

102 103 104 105 106


10−4

10−3

10−2

10−1

100

MARE

(b)


FIG. 6. Mean absolute relative error as a function of number of train-ing points. We show the error of the peak amplitud prediction as afunction of the number of training points for noise levels of (a) D = 0and (b) D = 10−4 ns−1, at µ = 2.2.

the number of nodes in the reservoir (D = 6000). Overall, theprediction of the amplitude of the upcoming chaotic pulse forµ = 2.45 requires less training points than for µ = 2.2. Theseresults suggest that the forecast of the dynamics with extremepulses is intrinsically harder to predict. It could also be thatthe low frequency of the extreme pulses makes them moredifficult to predict because they appear less frequently in thetraining set. However, they also appear less frequently in thetesting set and thus have less weight in the overall error.

Second, we analyze the influence of the stochastic contri-bution in Eq. 1 on the forecast of the pulses’ amplitude. Weshow in Figs. 6(b) and 7(b) that the presence of noise triggersan early plateau that bounds the MARE, deteriorating the per-formance of all the methods. The stochastic contribution tothe dynamics has a stronger influence on the forecast for thechaotic dynamics generated at µ = 2.45, with an increase oftwo orders of magnitude in the MARE as shown in Figs. 7 (a)and (b). When noisy dynamics is considered, the MARE forµ = 2.45 and µ = 2.2 are less than an order of magnitude apart

102 103 104 105 106


10−4

10−3

10−2

10−1

100

MARE

(a)


102 103 104 105 106


10−4

10−3

10−2

10−1

100

MARE

(b)


FIG. 7. Mean absolute relative error as a function of number of train-ing points. We show the error of the peak amplitud prediction as afunction of the number of training points for noise levels of (a) D = 0and (b) D = 10−4 ns−1, at µ = 2.45.

(see Figs. 6 (b) and 7(b)) in contrast to the noise-free coun-terparts for which the difference in MARE between µ = 2.45and µ = 2.2 is more apparent (see Figs. 6 (a) and 7(a)). Thedeterioration of the prediction accuracy in the presence of ob-servational noise has also been reported e.g. in16, where theNMSE decreased 5 to 6 orders of magnitude.

An important parameter for all but the reservoir computingapproach is the number of input intensity peak amplitudes (n)wherewith the machine learning algorithm is fed. This pa-rameter sets the amount of history that the algorithm is able to“see”. In Fig. 8, we show how the performance changes whenchanging n in the case of the chaotic dynamics with µ = 2.45and D = 10−4 ns−1. We used 10000 training data points tobe well inside the plateau of performance seen in Fig. 7(b).The results shown in this figure justifies our choice of usingn = 3, which yields a minimum MARE for most of the fore-cast methods. For the RC method, we set n = 1 as it is theonly method that possesses an internal memory.

Another important issue to consider when implementing

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755


0 2 4 6 8 10

Number of input peaks

10−3

10−2

10−1

100

MARE

Linear SVM

Gaussian SVM

Shallow NN

Deep NN

KNN

FIG. 8. Mean absolute relative error as a function of the number ofpreceding pulses fed as the input to each algorithm. For this example,all algorithms were trained using 10000 data points at µ = 2.45 andD = 10−4 ns−1. For both KNN and Deep NN the minimum error oc-curs when using 3 input peaks, being the latter the absolute minimumin this plot considering all other methods.

these data-driven methods is the computer power that is re-quired to train and test each forecast method. Although dif-ferent methods scale differently with the amount of data in thetraining set, some general rules of thumb apply. In the KNNmethod, while there is no specific training time, the time forevaluating each test point, however, grows linearly with theamount of points in the dataset. The KNN method is, in thissense, ideal for real-time data as it can take into account newdata into the dataset without any extra computational over-head. The computing power for training and testing the SVMmethods depends greatly on the amount of support vectors thatare needed, and on the kernel that is used. We find that thelinear SVM method takes a greater time to train and a compa-rable time to test w.r.t. the Gaussian kernel method. This isdue to the fact that the linear SVM method fails to capture thecomplexity of the data and, thus, a great amount of supportvectors are needed. The feed-forward neural networks andthe RC method are (in respect to train and test) opposite to theKNN method, they take a great amount of time to train butare computationally cheap to evaluate test points. The train-ing time of the neural networks-based models depends on thelength of the training data and on the amount of internal modelparameters they have. In our examples, the deep NN takesabout 20 times the time of the shallow NN to train. The RCmethod, on the other hand, has a simpler training mechanism,which takes approximately 5 times the time of the shallow NNto train. All neural networks-based models take a comparable(low) time in the testing stage.

We end up with a comparison of the performances obtainedhere with the literature. The reported MARE values that havebeen obtained strongly vary with the algorithm used and thecharacteristics of the datasets analyzed. For example, machinelearning techniques with delay embedding in real data giveMARE values of the order of 0.15 for river flow prediction24,or as low as 0.025 for electricity consumption14. With time se-

ries simulated from the chaotic Ikeda map, a MARE value aslow as 5.8×10−5 was reported in38. In general, the predictionof noisy (possibly chaotic) real-world dynamics yields largererrors than the prediction of synthetic numerical data withoutnoise. A more precise direct comparison of previously pub-lished results is, however, not currently possible since we donot predict the future trajectory of the dynamics but the am-plitude for the next pulse.

V. CONCLUSIONS

We have used the chaotic dynamics of the intensity of anoptically injected laser to test the performance of several ma-chine learning algorithms for forecasting the amplitude of thenext intensity pulse. This laser system is described by a sim-ple model that, with a small change of parameters, producestime series which have extreme events in the form of highpeak intensities, resembling the dynamics of much more com-plex systems. In spite of the fact that the autocorrelation func-tion of the sequence of pulse amplitudes decays rapidly, goodprediction accuracy was achieved with some of the proposedmethods, namely the KNN, Deep NN and RC methods. Wehave verified that the MARE for the most accurate methods(DNN, KNN and RC) remains approximately constant evenfor the prediction of extreme pulses that have a probability ofappearance as low as 1/1000.

Our work suggests that similar methods may be used in theforecast of more complex systems, although further testingis of course necessary to asses how well they would performin high-dimensional chaotic dynamical systems. While withsimple dynamics, we only needed around 1000 data pointsto achieve maximum performance with some methods (shal-low and deep NN); when forecasting more complex dynamics,some of the methods (KNN and deep NN) will continue im-proving their performance if longer datasets are available fortraining (longer than 105 data points).

We have also compared the performance of different vari-ations of the same machine learning algorithm (compare lin-ear to Gaussian SVM and shallow to deep neural networks inFigs. 4-7), especially relevant when considering big trainingdatasets. We do not exclude that even more complex methods(e.g. a neural network with additional hidden layers) mightoutperform the presented algorithms. However, the presentedalgorithms already serve the purpose of showing the depen-dence of the forecast error on the complexity of the dynamicsand on the inclusion of stochastic contributions.

ACKNOWLEDGMENTS

P. A. and C. M. acknowledge support by the BE-OPTICALproject (H2020-675512). C.M. also acknowledges supportfrom Spanish Ministerio de Ciencia, Innovación y Universi-dades (PGC2018-099443-B-I00) and ICREA ACADEMIA.M.C.S. was supported through a “Ramon y Cajal” Fellow-ship (RYC-2015-18140). This work is the result of a collab-oration established within the Ibersinc network of excellence

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755


(FIS2017-90782-REDT).1J. Ohtsubo, Semiconductor lasers: stability, instability and chaos, Vol. 111(Springer, 2012).

2S. Wieczorek, B. Krauskopf, T. B. Simpson, and D. Lenstra, “The dynami-cal complexity of optically injected semiconductor lasers,” Physics Reports416, 1–128 (2005).

3E. K. Lau, X. Zhao, H.-K. Sung, D. Parekh, C. Chang-Hasnain, and M. C.Wu, “Strong optical injection-locked semiconductor lasers demonstrating>100-ghz resonance frequencies and 80-ghz intrinsic bandwidths,” OpticsExpress 16, 6609–6618 (2008).

4K.-H. Lo, S.-K. Hwang, and S. Donati, “Numerical study of ultrashort-optical-feedback-enhanced photonic microwave generation using opticallyinjected semiconductor lasers at period-one nonlinear dynamics,” Opticsexpress 25, 31595–31611 (2017).

5C. Xue, S. Ji, A. Wang, N. Jiang, K. Qiu, and Y. Hong, “Narrow-linewidthsingle-frequency photonic microwave generation in optically injected semi-conductor lasers with filtered optical feedback,” Optics letters 43, 4184–4187 (2018).

6X.-Z. Li and S.-C. Chan, “Heterodyne random bit generation using an op-tically injected semiconductor laser in chaos,” IEEE Journal of QuantumElectronics 49, 829–838 (2013).

7N. Köllisch, J. Behrendt, M. Klein, and N. Hoffmann, “Nonlinear realtime prediction of ocean surface waves,” Ocean Engineering 157, 387–400(2018).

8C. Franzke, “Predictability of extreme events in a nonlinear stochastic-dynamical model,” Physical Review E 85, 031134 (2012).

9S. Birkholz, C. Brée, A. Demircan, and G. Steinmeyer, “Predictability ofrogue events,” Physical review letters 114, 213901 (2015).

10J. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott, “Model-free prediction oflarge spatiotemporally chaotic systems from data: A reservoir computingapproach,” Physical review letters 120, 024102 (2018).

11J. Isensee, G. Datseris, and U. Parlitz, “Predicting spatio-temporaltime series using dimension reduced local states,” arXiv preprintarXiv:1904.06089 (2019).

12S. Bialonski, G. Ansmann, and H. Kantz, “Data-driven prediction and pre-vention of extreme events in a spatially extended excitable system,” Physi-cal Review E 92, 042910 (2015).

13T. Kuremoto, S. Kimura, K. Kobayashi, and M. Obayashi, “Time se-ries forecasting using a deep belief network with restricted boltzmann ma-chines,” Neurocomputing 137, 47–56 (2014).

14J. Wang, D. Chi, J. Wu, and H.-y. Lu, “Chaotic time series method com-bined with particle swarm optimization and trend adjustment for electric-ity demand forecasting,” Expert Systems with Applications 38, 8419–8429(2011).

15M. Ardalani-Farsa and S. Zolfaghari, “Chaotic time series prediction withresidual analysis method using hybrid elman–narx neural networks,” Neu-rocomputing 73, 2540–2553 (2010).

16A. Gholipour, B. N. Araabi, and C. Lucas, “Predicting chaotic time seriesusing neural and neurofuzzy models: a comparative study,” neural process-ing letters 24, 217–239 (2006).

17K. Lau and Q. Wu, “Local prediction of non-linear time series using supportvector regression,” Pattern Recognition 41, 1539–1547 (2008).

18S. Perrone, R. Vilaseca, J. Zamora-Munt, and C. Masoller, “Controllingthe likelihood of rogue waves in an optically injected semiconductor laservia direct current modulation,” Physical Review A 89, 033804 (2014).

19N. Akhmediev, B. Kibler, F. Baronio, M. Belic, W.-P. Zhong, Y. Zhang,W. Chang, J. M. Soto-Crespo, P. Vouzas, P. Grelu, et al., “Roadmap on op-tical rogue waves and extreme events,” Journal of Optics 18, 063001 (2016).

20C. Bonatto, M. Feyereisen, S. Barland, M. Giudici, C. Masoller, J. R. R.Leite, and J. R. Tredicce, “Deterministic optical rogue waves,” Physicalreview letters 107, 053901 (2011).

21H. L. d. S. Cavalcante, M. Oriá, D. Sornette, E. Ott, and D. J. Gauthier,“Predictability and suppression of extreme events in a chaotic system,”Physical review letters 111, 198701 (2013).

22J. Zamora-Munt, B. Garbin, S. Barland, M. Giudici, J. R. R. Leite, C. Ma-soller, and J. R. Tredicce, “Rogue waves in optically injected lasers: Origin,predictability, and suppression,” Physical Review A 87, 035802 (2013).

23M. San Miguel and R. Toral, “Stochastic effects in physical systems,” inInstabilities and nonequilibrium structures VI (Springer, 2000) pp. 35–127.

24Z. He, X. Wen, H. Liu, and J. Du, “A comparative study of artificial neuralnetwork, adaptive neuro fuzzy inference system and support vector ma-chine for forecasting river flow in the semiarid mountain region,” Journalof Hydrology 509, 379–386 (2014).

25N. S. Altman, “An introduction to kernel and nearest-neighbor nonparamet-ric regression,” The American Statistician 46, 175–185 (1992).

26B. E. Boser, I. M. Guyon, and V. N. Vapnik, “A training algorithm foroptimal margin classifiers,” in Proceedings of the fifth annual workshop onComputational learning theory (ACM, 1992) pp. 144–152.

27V. Vapnik, The nature of statistical learning theory (Springer science &business media, 2013).

28T.-M. Huang, V. Kecman, and I. Kopriva, Kernel based algorithms formining huge data sets, Vol. 1 (Springer, 2006).

29H. Drucker, C. J. Burges, L. Kaufman, A. J. Smola, and V. Vapnik, “Sup-port vector regression machines,” in Advances in neural information pro-cessing systems (1997) pp. 155–161.

30Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E.Hubbard, and L. D. Jackel, “Handwritten digit recognition with a back-propagation network,” in Advances in neural information processing sys-tems (1990) pp. 396–404.

31D. Verstraeten, B. Schrauwen, M. d’Haene, and D. Stroobandt, “An exper-imental unification of reservoir computing methods,” Neural networks 20,391–403 (2007).

32A. Rodan and P. Tino, “Minimum complexity echo state network,” IEEEtransactions on neural networks 22, 131–144 (2010).

33M. Lukoševicius and H. Jaeger, “Reservoir computing approaches to re-current neural network training,” Computer Science Review 3, 127–149(2009).

34L. Larger, M. C. Soriano, D. Brunner, L. Appeltant, J. M. Gutiérrez, L. Pes-quera, C. R. Mirasso, and I. Fischer, “Photonic information processingbeyond turing: an optoelectronic implementation of reservoir computing,”Optics express 20, 3241–3249 (2012).

35Y. Paquot, F. Duport, A. Smerieri, J. Dambre, B. Schrauwen, M. Hael-terman, and S. Massar, “Optoelectronic reservoir computing,” Scientificreports 2, 287 (2012).

36Y. Kawai, J. Park, and M. Asada, “A small-world topology enhances theecho state property and signal propagation in reservoir computing,” NeuralNetworks 112, 15–23 (2019).

37A. Griffith, A. Pomerance, and D. J. Gauthier, “Forecasting chaotic sys-tems with very low connectivity reservoir computers,” arXiv:1910.00659(2019).

38Z. O. Yang, Y. Wang, D. Li, and C. Wang, “Predict the time series of theparameter-varying chaotic system based on reduced recursive lease squaresupport vector machine,” in 2009 International Conference on Artificial In-telligence and Computational Intelligence, Vol. 1 (IEEE, 2009) pp. 29–34.

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

0 1 2 3 4 5 6 7 8

Time lag (peaks)

-0.5

0

0.5

1

-0.5

0

0.5

1

-0.5

0

0.5

1

-0.5

0

0.5

1

Autocorrelation

0 2 4 6 8 10 12 14 16 18 20

Time (ns)

0

2

0

2

0

2

4

6

8

0

2

4

Intensity

(A.U

.)abcd

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

0 2 4 6

time (ns)

0

2

4

6

Intensity

Intensity signalLinear SVMGaussian SVMShallow NNDeep NNKNNRC

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

102 103 104 105 106


10−4

10−3

10−2

10−1

100

MARE

(a)


This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

102 103 104 105 106


10−4

10−3

10−2

10−1

100

MARE

(b)


This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

102 103 104 105 106


10−4

10−3

10−2

10−1

100

MARE

(a)


This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

102 103 104 105 106


10−4

10−3

10−2

10−1

100

MARE

(b)


This

is the

autho

r’s pe

er re

viewe

d, ac

cepte

d man

uscri

pt. H

owev

er, th

e onli

ne ve

rsion

of re

cord

will

be di

ffere

nt fro

m thi

s ver

sion o

nce i

t has

been

copy

edite

d and

type

set.

PLEA

SE C

ITE

THIS

ART

ICLE

AS

DOI:

10.10

63/1.

5120

755

0 2 4 6 8 10

Number of input peaks

10−3

10−2

10−1

100

MARE

Linear SVM

Gaussian SVM

Shallow NN

Deep NN

KNNTh

is is

the au

thor’s

peer

revie

wed,

acce

pted m

anus

cript.

How

ever

, the o

nline

versi

on of

reco

rd w

ill be

diffe

rent

from

this v

ersio

n onc

e it h

as be

en co

pyed

ited a

nd ty

pese

t.PL

EASE

CIT

E TH

IS A

RTIC

LE A

S DO

I: 10

.1063

/1.51

2075

5

Documents

Machine learning algorithms for predicting the amplitude ... · Machine learning algorithms for predicting the amplitude of chaotic laser pulses Machine learning algorithms for predicting