Upload
harshit-arora
View
225
Download
0
Embed Size (px)
8/3/2019 Desire Gettingstarted
1/30
David Scuse September, 20111
DESIRE NEURAL NETWORK SYSTEM
Introduction:
The Desire system is a neural network modelling tool that allows the user to build models ofneural networks. Using the system, almost any neural network model can be developed,including those that require differential equations. This is in contrast to conventional neuralnetwork tools that contain a variety of pre-defined models; the user can modify a modelsparameters but can not define new models (i.e. the system is a black box). With the Desiresystem, the neural network is defined using statements in the Desire programming language sothe user can see how each portion of the neural network performs its processing and, if necessary,can modify the processing. The Desire language is a high-level language so the user does not getlost in the details (as happens with neural network toolkits written in C/C++).
Running the DesireW (Windows) System
The Desire system (currently version 15.0) runs under Windows and Linux. (The followinginstructions refer to the Windows version.) To begin, double-click the command fileDesireW.bat to launch the Desire system. This batch file opens a Desire editor window and theDesire command window.
Figure 1: The Desire Editor Window and the Desire Command Window
8/3/2019 Desire Gettingstarted
2/30
David Scuse September, 20112
Load a file into the Editor window either by dragging and dropping it onto the editor windowor by using the editors Open command). You may store your files in the same folder as theDesire system or in any other convenient folder. Do not use Windows to associate Desire sourcefiles (.src and .lst) with the Desire Editor doing so causes the OK button in the Editorwindow to stop functioning correctly.
Figure 2: Loading A Desire Source File
8/3/2019 Desire Gettingstarted
3/30
David Scuse September, 20113
Transfer the file to Desire by clicking on the OK button in the editor window (or using the
shortcut alt-T O). Then type the command erun (or zz) into the Desire command window.This command causes the program to be executed.
Figure 3: Transferring A Desire Source File to the Desire Command Window
8/3/2019 Desire Gettingstarted
4/30
8/3/2019 Desire Gettingstarted
5/30
David Scuse September, 20115
Figure 5: Graph Window Menu Items
Similarly, the standard Windows facilities for copying and pasting the contents of the Desire
command window are available by clicking the top left icon of the window.
Figure 6: Command Window Menu Items
By modifying the Properties of the command window, you can switch from the default white texton a black background to black text on a white background.
Source Files
Normally, Desire source files have the extension .src. Files with the .src extension do not
contain line numbers while numbered source files have the extension .lst. If a source filealready contains line numbers, the line numbers can be removed by using the Desire commandwindow command keep 'xxx'to create an unnumbered version of the file on disk (the file willbe named xxx.src). Internally, Desire still keeps a file numbered so that it can generate errormessages and so that the programmer can refer to individual lines.
In general, unnumbered source files are preferable to numbered source files in Desire.
8/3/2019 Desire Gettingstarted
6/30
David Scuse September, 20116
To view the line numbers used in the Desire command window for the current source file, typethe Desire command list+. This causes the file to be listed with line numbers in the commandwindow. To list a specific line, type list+ 2200 (or, for a range of lines, type list+ 2200-2300) in the Desire command window. To save a copy of the current file with line numbers, typelist+ 'filename.lst' This causes a copy of the file to be saved in the Desire directory (which
is not necessarily the same directory as the original source file). If you want to save the .lstfilein a different directory, add the complete path to the file in the command; for example, list+'D:\MyFolder\filename.lst'
A Quick Tour: Learning the OR Patterns
The following sections briefly describe a simple single-layer, feedforward Desire neural network.The SLFF1 program in Figure 7 defines a neural network that attempts to learn the logical OR
patterns ({0,0} {0}, {1,1} {1}, etc.). The SLFF1 program defines a 2-1 network ofthreshold units and uses the delta training rule to learn the patterns. The output of the SLFF1program is shown in the Graph Window in Figure 8. Each of the generated symbols ( )
represents the value of the tss (the sum of squares of the error) at a particular point in time. Thescreen is organized in a graph format: the vertical axis represents the tss and ranges from -scale (represented by -) at the bottom to +scale (represented by +) at the top. The value of scaleis defined in the program and is displayed at the bottom of the screen. In this example, thevertical axis ranges from -4 at the bottom to +4 at the top. The horizontal axis represents thetime, t, and begins at 1 on the left and ranges to the total number of training iterations on theright. The number of iterations of the program is the number of training epochs times thenumber of patterns. A training epoch is the time period during which each pattern is presentedonce to the network. The number of training epochs is defined in the variable Nepochs. In theprogram below, the number of program iterations is 100 (25 * 4). The horizontal axis on thegraph displays the values of time, t, beginning at 1 and continuing for 100 iterations.
------------------------------------------------------ Single-Layer Feedforward Network (SLFF1)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1ARRAY Layer1[Ninp],Layer2[Nout],Weights[Nout,Ninp]ARRAY Target[Nout],Error[Nout]ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the OR training patterns--------------------------------------------------data 0,0;1,0;0,1;1,1 | read INPUTdata 0;1;1;1 | read TARGET---------------------------------------------------- Initialize Run-Time Variables
--------------------------------------------------scale=4 | display R | display N-16 | display W 362,0Lrate=0.2 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------tss=0t=1 | NN=Nepochs*Npat | TMAX=NN-1drunSTOP--------------------------------------------------DYNAMIC
8/3/2019 Desire Gettingstarted
7/30
David Scuse September, 20117
---------------------------------------------------- The following statements learn the weights-- The statements are executed once for each training pattern-- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=tVECTOR Layer1=INPUT#VECTOR Target=TARGET#VECTOR Layer2=swtch(Weights*Layer1)
VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1DOT SSQ=Error*Errortss=tss+SSQdispt tss---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE Npattss=0 | -- reset tss to zero for next epoch
Figure 7: SLFF1 OR Patterns Program
Figure 8: SLFF1 OR Patterns Program Output
8/3/2019 Desire Gettingstarted
8/30
David Scuse September, 20118
In Figure 8, the value of the tss increases as each pattern is processed during an epoch and it isdifficult to determine the actual system error by observing the graph. To simplify the graph, thetss is normally stored in a different variable (such as TSS) which is not modified until the end ofeach epoch.
------------------------------------------------------ Single-Layer Feedforward Network (SLFF1b)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1ARRAY Layer1[Ninp],Layer2[Nout],Weights[Nout,Ninp]ARRAY Target[Nout],Error[Nout]ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the OR training patterns--------------------------------------------------data 0,0;1,0;0,1;1,1 | read INPUTdata 0;1;1;1 | read TARGET---------------------------------------------------- Initialize Run-Time Variables--------------------------------------------------
scale=4 | display R | display N-16 | display W 362,0Lrate=0.2 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------
tss=0 | TSS=scale
t=1 | NN=Nepochs*Npat | TMAX=NN-1drunwrite TSSSTOP--------------------------------------------------DYNAMIC---------------------------------------------------- The following statements learn the weights and bias terms-- The statements are executed once for each training pattern-- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=t
VECTOR Layer1=INPUT#VECTOR Target=TARGET#VECTOR Layer2=swtch(Weights*Layer1)VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1DOT SSQ=Error*Errortss=tss+SSQ
dispt TSS
---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE Npat
TSS=tss
tss=0 | -- reset tss to zero for next epoch
Figure 9: SLFF1b OR Patterns Program
The program in Figure 9 displays the tss after the program has been modified to use the statementdispt TSS instead of dispt tss. Note that the name of the variable being graphed (tss inFigure 8 and TSS in Figure 10) appears at the bottom of the graph. As the number of epochsincreases, the graph of the TSS becomes smoother. The TSS is the value of the sum of the psssfor the previous epoch.
8/3/2019 Desire Gettingstarted
9/30
David Scuse September, 20119
Figure 10: SLFF1b OR Patterns Program Output
As can be seen in Figure 10, the TSS converges to 0, meaning that the patterns can be learned bythis network.
Learning the AND Patterns:
We can modify the program to use the AND patterns as shown below:
data 0; 0; 0; 1 | read TARGET | -- (AND patterns)
8/3/2019 Desire Gettingstarted
10/30
David Scuse September, 201110
If we attempt to learn the AND patterns, the following output is generated.
Figure 11: SLFF1b AND Patterns Program Output
For some reason, the AND patterns can not be learned with the same network.
Adding an additional input to the output unit makes it possible to learn the AND patterns. Thisadditional term is often referred to as a bias term. The bias term always has an input of1 andcontains an adjustable weight.
------------------------------------------------------ Single-Layer Feedforward Network (SLFF1c)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1
ARRAY Layer1[Ninp],Layer2[Nout],Bias[Nout],Weights[Nout,Ninp]
ARRAY Target[Nout],Error[Nout]ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the AND training patterns
--------------------------------------------------data 0,0; 0,1; 1,0; 1,1 | read INPUTdata 0; 0; 0; 1 | read TARGET---------------------------------------------------- Initialize Run-Time Variables--------------------------------------------------scale=4 | display R | display N-12 | display W 362,0Lrate=0.2 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------tss=0 | TSS=scalet=1 | NN=Nepochs*Npat | TMAX=NN-1
8/3/2019 Desire Gettingstarted
11/30
David Scuse September, 201111
drunwrite TSSSTOP
--------------------------------------------------DYNAMIC---------------------------------------------------- The following statements learn the weights and bias terms-- The statements are executed once for each training pattern
-- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=tVECTOR Layer1=INPUT#VECTOR Target=TARGET#
VECTOR Layer2=swtch(Weights*Layer1+Bias)
VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1
Vectr delta Bias=Lrate*Error
DOT SSQ=Error*Errortss=tss+SSQdispt TSS---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE NpatTSS=tss
tss=0 | -- reset tss to zero for next epoch
Figure 12: SLFF1c AND Patterns Program
As can be seen in the output below, the AND patterns can now be learned correctly.
Figure 13: SLFF1c AND Patterns Program Output
8/3/2019 Desire Gettingstarted
12/30
David Scuse September, 201112
The SLFF1 Program:
In this section, we examine the commands used in the various SLFF1 programs above.
The first statement in the program (excluding the initial comments) assigns initial values to thevariables Npat, Ninp, and Nout. These are simple variables (scalars) and the values assignedare simple numeric values. Note that all variable names are case sensitive and must be typedcorrectly.
The next set of statements allocates storage for the vectors and matrices that will be used by thesystem.
The data statement defines the values to be assigned to the INPUT matrix. Using the datastatement makes assigning values to a vector or matrix simpler than reading the values from afile or assigning values to the individual elements of a vector or matrix. The read statementcopies the values that are in the current data list into the named vector or matrix. Thus, theINPUT matrix contains 4 rows with 2 values per row (i.e. the 4 input patterns) and the TARGET
matrix contains 4 rows with 1 value per row (the 4 output patterns).
The display statement is used to set various Desire Graph Window display options. display Rgenerates thick dots on the screen while display Q generates thin dots. display N-x generatesblack lines on a white background; display Nx generates white lines on a black background.The value of x in these two statements identifies the starting colour for the graphs that aregenerated. The statement display W 362,0 defines the position of the top-left corner of thegraph window on the screen. This particular co-ordinate (362,0) works nicely with 1024x768resolution.
The next several statements assign values to scalar variables in preparation for the actual learning
process.
The drun statement causes the statements in the dynamic segment to be compiled and thenexecuted NN times. The two variables that control the number of times that the dynamicsegment is executed must have been initialized in the interpreted segment. The initial timevariable, t, is normally initialized to 1. The number of times that the dynamic segment is to beexecuted is defined in the variable NN which must also be initialized by the programmer in theinterpreted segment. The time variable t is automatically incremented by Desire by 1 each timethat the statements in the dynamic segment are executed until t becomes greater than NN. Oncethe statements in the dynamic segment have been executed the specified number of times (NN),the statement that follows the drun statement in the interpreted segment is executed. This
statement is normally a STOP statement.
In this program, the statements in the dynamic segment are executed 100 times. Note that mostof the statements in the dynamic segment of the program are preceded by one of the keywords:VECTOR, Vectr, MATRIX, or DOT. This indicates that the operation is performed on allelements of the associated vector or matrix.
The first statement in the dynamic segment, iRow=t, assigns the current iteration number to the
8/3/2019 Desire Gettingstarted
13/30
David Scuse September, 201113
variable iRow. This variable is used implicitly in the next statement to extract a training patternfrom the matrix of training patterns.
The statement VECTOR layer1=INPUT# extracts pattern number iRow from the set of trainingpatterns stored in the matrix INPUT and stores the pattern in the vector layer1. The pattern that
is extracted is pattern ((iRow-1 mod Npat)+1). In this example, the first time that the dynamicsegment is executed, the pattern in row 1 is extracted. The next time that the dynamic segment isexecuted, the pattern in row 2 is extracted. Executing the dynamic segment 4 times constitutesone training epoch, with each pattern begin processed once. Normally, NN (the number of timesthe dynamic segment is executed) is a multiple of the number of patterns (Npat).
The statement VECTOR Layer2=swtch(Weights*Layer1+Bias) computes the result ofmultiplying the matrix Weights by the current input pattern (Layer1) and then adding the Biasterm. The resulting vector is sent through the threshold function (swtch is a built-in function)and the output is then assigned to the output vector (Layer2). In this program, the output layerconsists of a single unit, but in general, the output layer may consist of any number of units.
The statement VECTOR Error=Target-Layer2 computes a vector (Error) which contains thedifference between the actual output (Layer2) and the desired output (Target).
The statement DELTA Weights=Lrate*Error*Layer1modifies the matrix Weights based on thedifference between the actual output and the desired output.
The statement Vectr delta Bias=Lrate*Error modifies the vector Bias based on thedifference between the actual output and the desired output. The statement DOTSSQ=Error*Errorcomputes the sum of the squares of the individual error terms.
The statement dispt TSSgraphs the current value of the variable TSS.
The statements that follow the SAMPLE Npat statement are executed only at the end of eachepoch (i.e. once every Npat times). In this program, the statements that follow the SAMPLEstatement first copy the current value of the tss into the variable TSS and then reset the variabletss back to zero for the next epoch.
TSS=tsstss=0 | -- reset tss to zero for next epoch
As can be seen in Figure 13, the TSS begins at scale, then is changed to 1, increases to 3,remains at 3 for another epoch, then decreases to 2, decreases to 1, and finally decreases to 0, andremains at 0. The TSS remains the same for 4 consecutive values of t (i.e., a training epoch)because the TSS variable is updated only at the end of each epoch.
The > (displayed in the command window) is the Desire prompt character; this character isdisplayed after the system has completed its processing and is waiting for the users nextcommand. To terminate the Desire system, type bye.
8/3/2019 Desire Gettingstarted
14/30
David Scuse September, 201114
X-OR Patterns:
We now attempt to learn the X-OR patterns. The only program modification required is:
data 0; 1; 1; 0 | read TARGET | -- (X-OR patterns)
The output is shown below.
Figure 14: SLFF1d X-OR Patterns Program Output
Even though we developed a more powerful network with the addition of bias terms, the X-ORpatterns can not be learned.
Perceptron Neural Network:
We now improve the neural network by initializing the weights and the bias terms to smallrandom values. Using random values may help the network reach a solution that it could notreach if the weights and the bias terms are initialized to 0. This neural network is referred to as aPerceptron.
8/3/2019 Desire Gettingstarted
15/30
David Scuse September, 201115
------------------------------------------------------ Single-Layer Feedforward Network (SLFF1e)-- activation function: threshold-- learning rule: delta learning--------------------------------------------------Npat=4 | Ninp=2 | Nout=1ARRAY Layer1[Ninp],Layer2[Nout],Bias[Nout],Weights[Nout,Ninp]ARRAY Target[Nout],Error[Nout]
ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout]---------------------------------------------------- Define the X-OR training patterns--------------------------------------------------data 0,0; 0,1; 1,0; 1,1 | read INPUTdata 0; 1; 1; 0 | read TARGET------------------------------------------------------------- Randomize the weight matrix-----------------------------------------------------------WFactor = 1for i=1 to Nout
for j=1 to NinpWeights[i,j] = ran() * WFactornext
Bias[i] = ran() *WFactornext
---------------------------------------------------- Initialize Run-Time Variables
--------------------------------------------------scale=4 | display R | display N-16 | display W 362,0Lrate=0.1 | Nepochs=25---------------------------------------------------- Learn the weights--------------------------------------------------tss=0 | TSS=scale | t=1 | NN=Nepochs*Npat | TMAX=NN-1drunSTOP
--------------------------------------------------DYNAMIC---------------------------------------------------- The following statements learn the weights and bias terms-- The statements are executed once for each training pattern-- for a total of Nepochs * Npat learning repetitions--------------------------------------------------iRow=t
VECTOR Layer1=INPUT#VECTOR Target=TARGET#VECTOR Layer2=swtch(Weights*Layer1+Bias)VECTOR Error=Target-Layer2DELTA Weights=Lrate*Error*Layer1Vectr delta Bias=Lrate*ErrorDOT SSQ=Error*Errortss=tss+SSQdispt TSS---------------------------------------------------- The following statements are executed once per training epoch-- (instead of once per training pattern)--------------------------------------------------SAMPLE NpatTSS=tsstss=0 | -- reset tss to zero for next epoch
Figure 15: SLFF1e X-OR PatternsPerceptron Program
8/3/2019 Desire Gettingstarted
16/30
David Scuse September, 201116
Figure 16: SLFF1e X-OR Patterns Perceptron Program Output
Unfortunately, even with random weights, the X-OR patterns can not be learned.
In the 1950s it was shown that a Perceptron network could not learn the X-OR patterns; this
result caused neural network research to slow down significantly; it wasnt until several decadeslater that researchers solved the X-OR problem and research began moving forward morerapidly.
8/3/2019 Desire Gettingstarted
17/30
David Scuse September, 201117
Desire Programming Statements
Program Segments:
Each Desire program consists of two types of segments: an interpreted experiment-protocolsegment and a compiled dynamic program segment. The experiment-protocol segmentconsists of statements that define the structure of a model and the models parameters. Thesestatements are interpreted. The dynamic program segment contains the difference and/ordifferential equations that define the behaviour of the model over time. These statements arecompiled. The experiment-protocol program statements are placed first in the program; thedynamic program statements follow the interpreted statements; a DYNAMIC statement separatesthe two program segments.
Some statements can be placed in either an interpreted segment or in a dynamic segment; otherstatements can be placed in one type of segment but not in the other type of segment. Forexample, array declarations can be placed only in an interpreted segment; assignment statementscan be placed in either type of segment; and most of the vector/matrix manipulation statementscan be placed only in dynamic program segments.
Statement Syntax:
The Desire syntax is straightforward. Statement lines are terminated by a carriage return. Forexample,
Lrate=0.5
Statement lines may contain several statements, using the | character as a statement separator.
Lrate=0.5 | Yrate=1.3
The characters -- indicate that a statement is a comment (note that the blank is required).
-- this is a comment
If a comment is defined on the same line as another statement, the preceding statement must beterminated using the | statement separator. For example,
Lrate=0.5 | -- define the learning rate
Desire is case sensitive; the two names Lrate and lrate refer to different variables. Some systemcommands (such as the STOP command) must be in upper case. Symbols must start with a letterand may have up to 20 upper-case and/or lower-case alphanumeric characters and $ signs.
8/3/2019 Desire Gettingstarted
18/30
David Scuse September, 201118
Interpreter Statements:
Interpreter statements are translated line-by-line as they are executed. Interpreter statements maybe typed directly into the Desire system window by the user or they may be stored in a file forsubsequent execution. Interpreter statements that are typed directly into Desire are referred to as
commands. Interpreter statements that are typed into a file are referred to as programmedstatements.
The command-mode statement:
> Lrate=0.5
causes the variable Lrate to be assigned the value 0.5. This statement is executed (interpreted) assoon as the statement is typed. If the variable Lrate had been given a value earlier in the session,that value is replaced by this new value.
Compiled Statements:
Desire also contains statements that are compiled for quick execution. These statements have thesame syntax as interpreted statements but compiled statements can be entered only in a program.
System Files:
Desire uses several types of files; the file type is identified by the file extension. Files with anextension of .lstare ascii source files that include line numbers; files with an extension of .srcare ascii source files that do not include line numbers. The currently active source file isautomatically saved in syspic.lst.
Screen Input/Output:
The disptstatement is a compiled statement that can be used only in a dynamic segment. It isused to plot the values of one or more variables against the time variable, t.
The write statement is an interpreter statement that is used to display the current value of one ormore variables. For example, the statement:
write Weights | -- interpreter statement
causes the current values of the Weights matrix to be displayed on the screen. This statement canbe used inside a program (in an interpreted statement) or outside of a program (in command
mode). This statement is useful when testing built-in functions in command mode. For example,the following statement displays the result of evaluating the swtchfunction.
write swtch(0.0)
While the write statement can not be used in a dynamic segment, there is a correspondingstatement, the typestatement, that provides the same facility in dynamic segments.
8/3/2019 Desire Gettingstarted
19/30
David Scuse September, 201119
type X,Y,Z | -- dynamic statement
File Input/Output:
The interpreter write statement can also be used to write information to a disk file. For example,the interpreter statements:
connect 'temp.dat' as output 4write #4,Lratedisconnect 4
open a file for output, write the contents of the variable Lrateto the file, and then close the file.
Drun and Go Statements
Once a program has been run once, the programs dynamic statements can be executed againfrom where the previous execution left off by typing the drunstatement.
If there are executable statements that follow the STOPstatement in the interpreted program, thesestatements can be executed by typing the gostatement.
Interpreter Loops
A loop may be added to the interpreter program using the for statement. For example, thefollowing statements:
for count=1 to 10drunnext
cause the drun statement to be executed 10 times (the screen is cleared at the beginning of eachdrun). To examine the screen before it is cleared, a STOPstatement can be added to the loop.
for count=1 to 10drunSTOPnext
The system stops when the STOP statement is encountered; however, the user can issue the gostatement to instruct the system to resume its processing.
Display Statement
The display statement is used to set various Graph window properties. By default, thebackground of the Graph window is black. It is usually a good idea to switch to a whitebackground (particularly if the graph window is to be printed).
display N12 | -- use a black background and set initialcolour for graph to 12
display N18 | -- use a black background and use whitefor all graphs
8/3/2019 Desire Gettingstarted
20/30
David Scuse September, 201120
display N-12 | -- use a white background and set initialcolour for graph to 12
display N-18 | -- use a white background and use blackfor all graphs
display C16 | -- set graph axes to blackdisplay R | -- generate thick dots on graphdisplay Q | -- generate thin dots on graph
display W x,y | -- set position on screen of graph windowdisplay 2 | -- dont erase graph after drun (must be
placed after drun statement)
The display colour is used to identify each function that is graphed in the Graph window. In thelower portion of the graph window shown below, the Sig function is graphed using the firstcolour (the colour specified by x in the display Nx statement), the SigP function is graphedusing the next smaller colour (the colour x-1), the TanH function is graphed using the nextsmaller colour (the colour x-2), and so on. (This graph window is shown in its entirety in Figure18.)
Programmer-Defined Functions
The programmer may define functions at the beginning of the Desire program. For example, thefollowing function returns the minimum of two values:
FUNCTION min(aaa,bbb)=aaa-lim(aaa-bbb)
The following function returns a random integer between 1 and XX.
FUNCTION Rand(XX)=round((((ran()+1.0)/2.0)*XX)+0.5)
In the examples above, lim(x), ran(), and round(x) are Desire built-in functions. It isimportant that the parameters used in programmer-defined functions be variable names that areunique within the program (hence the choice of variables such as aaa and bbb).
Labeled Dynamic Segments
The first dynamic segment does not have a label; this segment is executed by invoking the drunstatement. Additional dynamic segments may also be defined; for example, the first dynamic
segment could train a network; a second segment could test the ability of the network to recall aset of patterns. Labeled dynamic segments are invoked using the drun name statement. Alabeled dynamic segment is identified using a label name statement at the beginning of thesegment.
CLEARN Statement
The CLEARN statement is used during competitive learning.
8/3/2019 Desire Gettingstarted
21/30
David Scuse September, 201121
CLEARN Output = Weights(Input)Lrate,Crit
where:Input is the input/pattern vectorWeights is the weight matrixLrate is the learning rate: if Lrate=0, there is no learning, just recallCrit is a flag: for Crit=0, a conscience mechanism is implemented;
for Crit
8/3/2019 Desire Gettingstarted
22/30
David Scuse September, 201122
Desire Graph Window
Graphing Functions
The following graphprogram illustrates how Desire can be used to graph various functions. Inthe program, the sigmoid, sigmoid-prime (first derivative of the sigmoid function), tanh(hyperbolic tangent), and tanh-prime (first derivative of tanh) are graphed.
----------------------------------------------------- This program graphs a variety of functions (graph1.src)---------------------------------------------------display N-12 | display Rscale=1tmin=-4.0 | incr=0.001 | tmax=4.0t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------
Sig=sigmoid(t) | -- sigmoid functionSigP=Sig*(1-Sig) | -- first derivative of sigmoid functionTanh=tanh(t) | -- tanh functionTanhP=1-Tanh*Tanh | -- first derivative of tanh functiondispt Sig,SigP,Tanh,TanhP
Figure 17: Graph Program
Figure 18: Graph Program Output
8/3/2019 Desire Gettingstarted
23/30
David Scuse September, 201123
Offset Graphs
At times, the graph window can become overloaded with functions that overlap, making itdifficult to determine which function is which (and it doesnt help that Desire only graphs the lastfunction if multiple function points occur at the same location on the graph). To make it easier to
understand graphs, it is often helpful to offset one or more of the graphs.
For example, the following program graphs 3 functions but the last function graphed erasesportions of the previous functions.
----------------------------------------------------- This program graphs the sigmoid function (graph2.src)---------------------------------------------------scale=1display N-6 | display Rtmin=-10.0 | incr=0.1 | tmax=10.0 | Shift=5t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------sig1=sigmoid(t+Shift)sig2=sigmoid(t)sig3=sigmoid(t-Shift)dispt sig1,sig2,sig3
Figure 19: Sigmoid Graph Program
Figure 20: Sigmoid Graph Program
8/3/2019 Desire Gettingstarted
24/30
David Scuse September, 201124
The following program graphs the same functions but offsets two of the functions by a portion ofthe value ofscale. As a result, the graphs become much easier to read (as long as you rememberthat they are offset).
----------------------------------------------------- This program graphs the sigmoid function
-- but offsets two of the graphs (graph3.src)---------------------------------------------------scale=1display N-12 | display Rtmin=-10.0 | incr=0.1 | tmax=10.0 | Shift=5t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------sig1=sigmoid(t+Shift)sig2=sigmoid(t)-scale/2sig3=sigmoid(t-Shift)-scaledispt sig1,sig2,sig3
Figure 21: Sigmoid Offset Graph Program
Figure 22: Sigmoid Offset Graph Program Output
8/3/2019 Desire Gettingstarted
25/30
David Scuse September, 201125
Reducing the Learning Rate
The following program illustrates how the learning rate can be gradually reduced. The top partof the graph displays the learning rate as it is reduced from its initial value of L0 (0.75) to itsfinal value of Ln (0.01). The function 1-sigmoid() (shown in the bottom half of the graph)
defines the transition from the initial learning rate to the final learning rate. Other functionscould be used to define the transition (such as a simple linear function) but the sigmoid functionprovides more time at both extremes than linear functions. Changing the parameters of thesigmoid function (S0 and Sn) would change the rate of transition of the learning rate.
------------------------------------------------------------- Reduce the learning rate (Lrate.src)-----------------------------------------------------------display R | display N-13 | display W 362,0scale=1Nepochs=1000 | Npat=1LRate0=0.75 | LRateN=0.01S0=-6 | SN=6 | SDelta=(SN-S0)/(Nepochs*Npat) | S=S0t=1 | NN=Nepochs | TMAX=NN-1drun
STOP-----------------------------------------------------------DYNAMIC-----------------------------------------------------------Sig=1-sigmoid(S)LRate=(LRate0-LRateN)*Sig+LRateNSig= Sig-scaleS=S+SDeltadispt LRate,Sig
Figure 23: Reducing the Learning Rate Program
Figure 24: Reducing the Learning Rate Program Output
8/3/2019 Desire Gettingstarted
26/30
David Scuse September, 201126
Clipping a Function
The following program illustrates how a function may be clipped to ensure that it fits withinthe specified value ofscale. (This prevents the situation where a function may be outside of thebounds ofscale and thus not appear anywhere in the Graph window.)
------------------------------------------------------------- Clip a function so that it fits within the graph (clipping.src)-----------------------------------------------------------FUNCTION min(aaa,bbb)=aaa-lim(aaa-bbb)FUNCTION max(aaaa,bbbb)=aaaa+lim(bbbb-aaaa)FUNCTION clipMax(X1)=min(X1,scale)FUNCTION clipMin(Y1)=max(Y1,-scale)FUNCTION clip(Z1)=clipMin(clipMax(Z1))-----------------------------------------------------------display N-12 | display Rscale=1tmin=-2.0 | incr=0.001 | tmax=2.0t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1drunSTOP---------------------------------------------------DYNAMIC
---------------------------------------------------T=1.2*tanh(t)clipT=clip(T)dispt clipT
Figure 25: Clipping a Function Program
Figure 26: Clipping a Function Program Output
8/3/2019 Desire Gettingstarted
27/30
David Scuse September, 201127
Graphing the Sigmoid Function with Temperatures
----------------------------------------------------- This program graphs the sigmoid function-- for several temperatures (graph4.src)---------------------------------------------------display N-13 | scale=1
tmin=-6.0 | incr=0.01 | tmax=6.0T1=4.0 | T2=0.25t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------Sig=sigmoid(t)Sig4=sigmoid(t/T1)Sig14=sigmoid(t/T2)dispt Sig,Sig4,Sig14
Figure 27: Sigmoid with Temperatures Program
Figure 28: Sigmoid with Temperatures Program Output
8/3/2019 Desire Gettingstarted
28/30
David Scuse September, 201128
Graphing Gaussian Bumps
----------------------------------------------------- This program graphs Gaussian bumps (graph5.src)---------------------------------------------------display N-13 | display Rscale=1
tmin=-4.0 | incr=0.01 | tmax=4.0s1=0.5 | s2=1.0 | s3=2.0t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1x=0.25drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------g05=exp(-(t*t)/(2*s1*s1))g10=exp(-(t*t)/(2*s2*s2))g20=exp(-(t*t)/(2*s3*s3))dispt g05,g10,g20
Figure 29: Gaussian Bumps Program
Figure 30: Gaussian Bumps Program Output
8/3/2019 Desire Gettingstarted
29/30
David Scuse September, 201129
Early Termination of a Dynamic Segment
The Desire system normally executes each dynamic segment until completion. However, thereare times when it is useful to be able to stop a dynamic segment before all NN repetitions arecomplete. (For example, when the tss has reached an acceptable value.) The termstatement can
be used in a dynamic segment for this purpose. As soon as the value of the expression in the termstatement becomes positive, processing of the dynamic segment is terminated.
----------------------------------------------------- This program terminates a dynamic segment early (term1.src)---------------------------------------------------display N-12 | display Rscale=1tmin=-4.0 | incr=0.01 | tmax=4.0t=tmin | TMAX=(tmax-tmin) | NN=TMAX/incr+1stop=2drunSTOP---------------------------------------------------DYNAMIC---------------------------------------------------term (t-stop)+0.0000001 | -- stop when this expression becomes positiveSig=sigmoid(t)
dispt Sig-- type t
Figure 31: Early Termination of a Dynamic Segment
Figure 32: Early Termination of a Dynamic Segment Program Output
8/3/2019 Desire Gettingstarted
30/30
id S S b 2011
Differential Equations
The DIFF program graphs the sigmoid and sigmoid-prime functions. It also illustrates the abilityof Desire to solve differential equations (numerically). A differential equation is introducedusing the keyword d/dt. (In addition to defining scalar differential equations, differential
equations may also be defined using vectors and matrices.) In this example, the first derivativeof Y is defined as Sig*(1-Sig); thus, Y is the sigmoid function (which Desire calculatescorrectly). Note that the sigmoid and sigmoid derivative have been shifted so that they appear inthe bottom half of the graph.
------------------------------------------------------------------- Working with a scalar differential equation (diff.src)-----------------------------------------------------------------display N-12 | display C16 | display Rscale=1tmin=-8.0 | incr=0.001 | tmax=8.0t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1drunSTOP-----------------------------------------------------------------
DYNAMIC-----------------------------------------------------------------Sig=sigmoid(t) | -- sigmoid functionSigP=Sig*(1-Sig) | -- first derivative of sigmoid functiond/dt Y = SigP | -- compute Y such that Y's first derivative is SigPerr = abs(Y) - abs(Sig)Sig=Sig-scaleSigP=SigP-scaledispt Sig,SigP,Y,err
Figure 33: Differential Equation Program
Figure 34: Differential Equation Program Output