Embed Size (px)
Citation preview
L. Manevitz U. Haifa 1
Neural Networks: Capabilities and ExamplesL. Manevitz
Computer Science Department
HIACS Research Center
University of Haifa
L. Manevitz U. Haifa 2
What Are Neural Networks?What Are They Good for?How Do We Use Them?
• Definitions and some history
• Basics– Basic Algorithms
– Examples
• Recent Examples
• Future Directions
L. Manevitz U. Haifa 3
Natural versus Artificial Neuron
• Natural Neuron McCullough Pitts Neuron
L. Manevitz U. Haifa 4
Definitions and History
• McCullough –Pitts Neuron
• Perceptron
• Adaline
• Linear Separability
• Multi-Level Neurons
• Neurons with Loops
L. Manevitz U. Haifa 5
Sample Feed forward Network (No loops)
•Weights •Weights
•Weights
•Input
•Output
•Wji•Vik
F(wji xj
L. Manevitz U. Haifa 6
Replacement of Threshold Neurons with Sigmoid or Differentiable Neurons
•Threshold •Sigmoid
L. Manevitz U. Haifa 7
Reason for Explosion of Interest
• Two co-incident affects (around 1985 – 87)
– (Re-)discovery of mathematical tools and algorithms for handling large networks
– Availability (hurray for Intel and company!) of sufficient computing power to make experiments practical.
L. Manevitz U. Haifa 8
Some Properties of NNs
• Universal: Can represent and accomplish any task.
• Uniform: “Programming” is changing weights
• Automatic: Algorithms for Automatic Programming; Learning
L. Manevitz U. Haifa 9
Networks are Universal
• All logical functions represented by three level (non-loop) network (McCullough-Pitts)
• All continuous (and more) functions represented by three level feed-forward networks (Cybenko et al.)
• Networks can self organize (without teacher).
• Networks serve as associative memories
L. Manevitz U. Haifa 10
Universality
• McCullough-Pitts: Adaptive Logic Gates; can represent any logic function
• Cybenko: Any continuous function representable by three-level NN.
L. Manevitz U. Haifa 11
Networks can “LEARN” and Generalize (Algorithms)
• One Neuron (Perceptron and Adaline) Very popular in 1960s – early 70s– Limited by representability (only linearly separable
• Feed forward networks (Back Propagation)– Currently most popular network (1987 –now)
• Kohonen self-Organizing Network (1980s – now)(loops)
• Attractor Networks (loops)
L. Manevitz U. Haifa 12
Learnability (Automatic Programming)
• One neuron: Perceptron and Adaline algorithms (Rosenblatt and Widrow-Hoff) (1960s –now)
Feed forward Networks: Backpropagation (1987 – now)
Associative Memories and Looped Networks (“Attractors”) (1990s – now)
L. Manevitz U. Haifa 13
Generalizability
• Typically train a network on a sample set of examples
• Use it on general class
• Training can be slow; but execution is fast.
L. Manevitz U. Haifa 14
•Pattern Identification
•(Note: Neuron is trained)
•weights
field receptivein threshold Axw ii kdkdkfjlll
field. receptive in the is letter The Axw ii
Perceptron
L. Manevitz U. Haifa 15
•weights
field receptivein threshold Axw ii kdkdkfjlll
Feed Forward Network
•weights
L. Manevitz U. Haifa 16
Classical Applications(1986 – 1997)
• “Net Talk” : text to speech
• ZIPcodes: handwriting analysis
• Glovetalk: Sign Language to speech
• Data and Picture Compression: “Bottleneck”
• Steering of Automobile (up to 55 m.p.h)
• Market Predictions
• Associative Memories
• Cognitive Modeling: (especially reading, …)
• Phonetic Typewriter (Finnish)
L. Manevitz U. Haifa 17
Neural Network
• Once the architecture is fixed; the only free parameters are the weights
• Thus Uniform ProgrammingUniform Programming
• Potentially Potentially Automatic ProgrammingAutomatic Programming
• Search for Learning AlgorithmsSearch for Learning Algorithms
L. Manevitz U. Haifa 18
Programming: Just find the weights!
• AUTOMATIC PROGRAMMING
• One Neuron: Perceptron or Adaline
• Multi-Level: Gradient Descent on Continuous Neuron (Sigmoid instead of step function).
L. Manevitz U. Haifa 19
Prediction
•Input/Output •NN
•delay
•Compare
L. Manevitz U. Haifa 20
Training NN to Predict
L. Manevitz U. Haifa 21
Finite Element Method
• Numerical Method for solving p.d.e.s
• Many user chosen parameters
• Replace user expertise with NNs.
L. Manevitz U. Haifa 22
FEM Flow chart
L. Manevitz U. Haifa 23
Problems and Methods
L. Manevitz U. Haifa 24
Finite Element Method and Neural Networks
• Place mesh on body
• Predict where to adapt mesh
L. Manevitz U. Haifa 25
Placing Mesh on Body (Manevitz, Givoli and Yousef)
• Need to place geometry on topology
• Method: Use Kohonen algorithm
• Idea: Identify neurons with FEM nodes
– Identify weights of nodes with geometric location
– Identify topology with adjaceny
– RESULT: Equi-probably placement
L. Manevitz U. Haifa 26
Kohonen Placement for FEM
• Include slide from Malik’s work.
L. Manevitz U. Haifa 27
Self-Organizing Network
•Weights from input to neurons
•Topology between neurons
L. Manevitz U. Haifa 28
Self-Organizing Network
•Weights from input give “location” to neuron
•Kohonen algorithm results in “winner” neuron
•After training, close input patterns have topologically close winners
•Results in Equi-probable Continuous
Mapping (without teacher)
L. Manevitz U. Haifa 29
Placement of Mesh via Self Organizing NNs
L. Manevitz U. Haifa 30
Placement of Mesh via Self Organizing NNs2
Iteration 0 Iteration 500;Quality =288
Iteration 2000;Quality = 238
Iteration 6000;Quality =223
Iteration 12000;Quality = 208
Iteration 30000;Quality =202
L. Manevitz U. Haifa 31
Comparison of NN and PLTMG
PLTMG (249 nodes) NN (225 nodes); Quality = 27922 )2()2(),( where),( yx
yyxx eeyxuuuyxf
Node
Error
Value
Error
Pltmg 2.4 E-02 4.51 E-02
NN 7.5 E-03 9.09E-03
L. Manevitz U. Haifa 32
FEM Temporal Adaptive Meshes
L. Manevitz U. Haifa 33
Prediction of Refinement of Elements
• Method simulates time
• Current adaptive method uses gradient
• Can just MISS all the action.
• We use NNs to PREDICT the gradient.
• Under development with Manevitz, Givoli and Bitar.
L. Manevitz U. Haifa 34
Training NN to Predict2
L. Manevitz U. Haifa 35
Refinement Predictors
•Need to choose features
•Need to identify kinds of elements
L. Manevitz U. Haifa 36
Other Predictions?
• Stock Market (really!)
• Credit Card Fraud (Master Card, USA)
L. Manevitz U. Haifa 37
Surfer’s Apprentice Program
• Manevitz and Yousef
• Make a “model” of user for retrieving information from internet.
• Many issues: here focus on retrieval of new pages similar to other pages of interest to user. Note ONLY POSITIVE DATA.
L. Manevitz U. Haifa 38
L. Manevitz U. Haifa 39
Bottleneck Network
•Train to Identity on Sample Data
•Should be identity only on similar data
•NOVELTY FILTER
L. Manevitz U. Haifa 40
How well does it work?
• Tested on Standard Reuter’s Data Base.
• Used 25% for training
• Withholding information on representation
• The best method for retrieval using only positive training. (Better than SVM, etc.)
L. Manevitz U. Haifa 41
How to help Intel? (Make Billions? Reset NASDAQ)
• Branch prediction?
• (Note similarity to FEM refinement.)
• Perhaps can use to give predictor that is even user or application dependent.
• (Note: Neural activity is, I am told, natural for VLSI design and there have been several such chips produced.)
L. Manevitz U. Haifa 42
Other Different Directions
• Modify basic model to handle temporal adaptivity. (Occurs in real neurons according to latest biological information.)
• Apply to model human diseases, etc.
