EE538-01.pdf

EE 538 Neural Networks

Soo-Young LeeDepartment of Electrical Engineering /

Brain Science Research CenterKAIST

EE538 Neural Networks Fall 2015 1-1

Lecture 1 Introduction Biological Neuron Models

Course Objectives• Understanding engineering models (artificial

neural networks / machine learning) of cognitive functions for “smart” machine– Feature extraction– Clustering– Separation– Classification/Recognition– Prediction– Motor control– and more


Related Topics


Rule-basedSystems

Learning-based Systems

Artificial Intelligence

Big Data/Data Mining

Machine Learning

Neural Networks (Connectionist Model)

Probabilistic Learning

KnowledgeDevelopment

LanguageUnderstanding Pattern/Image

RecognitionSpeech

Recognition

Course Emphasis• Connections between biological and artificial neural

networks– Related to computational models of brain functions

(Computational Neuroscience, not to heuristic rules)– Learn from hints

• Term-Project– Utilize neural network models to benchmark/competition

problems

EE538 Neural Networks 1-4

Approaches• Lecture• Homework (20%) and Quiz (10%)• Mid-Term Exam (20%)• Course Participation (10%)• Term Project (Proposal 15%, Report 35%)

– Repeat what others had done– Add your own ideas


Contents• Background• Course Overview• Biological Neural Networks • Artificial Neural Networks


International CES• 3D• Bigger• More pixels• Faster• Smaller• Distinct color

• SMART, SMART, and SMART!

EE538 Neural Networks 1-7 EE538 Neural Networks 1-8

EE538 Neural Networks 1-9 EE538 Neural Networks 1-10


Historical Sketch• Dankoon and Pygmalion• Pre-1940: von Hemholtz, Mach, Pavlov, etc.

– General theories of learning, vision, conditioning– No specific mathematical models of neuron operation

• 1940s: Hebb, McCulloch and Pitts– Mechanism for learning in biological neurons– Neural-like networks can compute any arithmetic

function• 1950s: Rosenblatt, Widrow and Hoff

– First practical networks and learning rules


Historical Sketch• 1960s-1970s: Dark age

– Minsky and Papert demonstrated limitations of artificial neural networks

– Progress continues, although at a slower pace • Mid 1980s: First Renaissance

– Important new developments caused a resurgence• Late 1980s - Mid 2010s: Continuing Developments

– New algorithms and applications• Mid-2010s – Today : Second Renaissance

– Smart, Smart, and Smart applications!– Big data (in Internet and mobile networks) and cheap

multicore processors (GPUs)EE538 Neural Networks 1-13

Sen-sor

FeatureExtraction

Recogn./Tracking

SelectiveAttention

Sen-sor

FeatureExtraction

Recog./Understand

SelectiveAttention

Perception/Planning

Touch Olf. Smell

Inference

BodyControl

Individual/Group

Behavior

Learning/Memory/Language

Fusion

Pattern Recognition

Speech Recognition

Cognition & Inference

Brain-like SystemBehavior

Vision

Auditory


Korean BrainTech21 Program (Brain Engineering / Neuroinformatics: 1998-2008)

Active Cognitive Development and Situation Awarenessbased on Cognitive Science and Information Technology

AudioVideo

Artificial Cognitive Systems (2009-2014)


Example: Auditory Processing

Feature Extraction

Sound Localization & Speech Enhancement(Separation)

Top-Down Attention

特徵抽出

音源探知 /音聲向上

下向注意集中

Classification/RecognitionPrediction


ICA Features from Natural Images


Speech Features at Cochlea (J.H.Lee, et.al., 2000)

Time Domain

Frequency Domain

ICA on speech signals (x=c1f1+c2f2+…+cMfM)Gabor-like features both in time and frequency domain

Frame


ICA-based Complex Features(T. Kim, et.al., 2005)

Time Onset/Offset

Multifrequency Tone(Timbre)

Frequency Modulation

Time Frame

Mel-

Fre

quency

Each training sample


Signal Separation (T. Kim, et al.)

• Recorded microphone signals

• Separated output signals

• Recording environment– Recorded in reverberant real

conference room – Sampling rate: 8 kHz (down

sampled from 44.1 kHz)– 3 human speakers are speaking

and 1 loud speaker is playing a hip-hop music simultaneously

(.wav) (.wav)

(.wav)

(.wav) (.wav)

(.wav)


Classification/Recognition• Speech Recognition

• Image Recognition


Time Frame

Mel-

Fre

quency

Each training sample

Top-Down Attention• Training Patterns (32x32 pixels)

• Deformed Test Patterns


Prediction• Stock price prediction


Motor Control


Contents• Background• Course Overview• Biological Neural Networks • Artificial Neural Networks


Neural Network ModelsWe need introduce • Neuron models• Network architectures • Learning algorithms

– Mathematical models• Applications to engineering problems

– Signal/image processing and recognition– Language understanding and knowledge

developmentof artificial neural networksEE538 Neural Networks 1-26

Neural Networks

Cognitive Science

/NeuroscienceIT (EECS)

Biologically-Inspired Electronic SystemsDigital Brain (Neural Networks)

Biologically-Oriented Electronic SystemsNeuroinformatic DB, BioMedical Imaging

Bionic Life

EE538 Neural Networks 1-27 BiS554 Neural Networks

Top of

the World


Research Scope

Systems

Molecules

Data Mathematical Applications/Measurement Modeling Implementation

GenProtein

MembraneNeuronCircuitsSystem

BehaviorNeural

Networks


Research Modality

Neuroscience Data

Mathematical Model

Brain-like Functional Systems

Neuromorphic Chips

Analysis Software

Measurement Technology


Research Tools• Mathematics

– Optimization– Linear algebra– Differential equation– Statistics / Information theory

• Computer Programming• Hardware Devices

– Silicon-based VLSI– MEMS


Contents• Background• Course Overview• Biological Neural Networks• Artificial Neural Networks


Brain vs. Computer• Human Brain

– about 10^10 to 10^11 neurons– about 10^3 to 10^4 synapses for each neuron– about 10^14 synapses – about 1.5 kg (2% of body weight)– about 1-10 msec time scale

• While computers just conduct programmed instructions, human brain interact with and learns from environments.


Cortical Parameters


Neuron andSynapses


Different Types of Ion Channels


Ion Channels and Concentrations


ActionPotential



Contents• Background• Course Overview• Biological Neural Networks• Artificial Neural Networks


Neural Network Models• Neuron model

– Integrate-and-fire– Saturating non-linearity– Rectified Linear Unit

• Network architecture– Feedforward

• Layered, Convolutive, Max-Pooling– Recurrent

• Learning algorithm– Unsupervised learning– Supervised learning– Reinforcement learning


How to Conduct Researches ? • Set hypothesis, make model, and evaluate the model.

– Hypothesis: best educated guess• Knowledge on neuroscience

– Model: network architecture, mathematical equations

• Knowledge on mathematics (linear algebra, statistics, differential equations, etc.)

– Evaluation• Experiment and/or simulation• Find parameters• Performance measure and decision criteria


Example: Neuron Model• Hypothesis:

– Interaction between membrane potential and ion concentration with voltage-dependent ion channels

• Model: Hodgkin-Huxley model• Evaluation

– Voltage clamp experiment– Find parameters


Action Potential


Hodgkin-Huxley model


Basic BioPhysics• Fick’s law: diffusion

• Ohm’s law: drift

• Einstein relationship

• Equllibriumq

kTD

dxdnDJdiffusion

dxdVZnJdrift

0

dxdVZn

dxdn

qkTJJJ driftdiffusion


Nernst Potential

i

on

n

V

Voi n

nqZkTdn

nqZkTdVVVE

i

o

i

o

ln1

• Nernst potential


Ion Parameters


Resting Potential (1)• Goldman equation for multi-ions• For one-type ions

where

nV

kTqZ

dxdnP

dxdVZn

dxdn

qkTJJJ mdriftdiffusion

D

qkTPV

dxdV m ,

dnn

qZPVkTJqZV

kTdnnPV

kTqZJ

Pdxi

o

i

o

n

n

m

m

n

nm

11,1

0

)/exp(1)/exp(,lnkTZqV

nkTZqVnPVkTqZJ

nqZPV

kTJ

nqZPV

kTJ

kTqVZ

m

omim

im

omm


Resting Potential (2)• For Potassium and Chlorine ions

• Goldman equation for Potassium and Chlorine ions

• Goldman equation for Potassium, Sodium, and Chlorine ions ?

• Do we need any other ion transfer mechanism?

)/exp(1][)/exp(][

)/exp(1][)/exp(][

kTqVClkTqVCl

kTPqVJ

kTqVKkTqVK

kTPqVJ

m

omiClmCl

m

omiKmK

oCliK

iCloKm ClPKP

ClPKPq

kTV][][][][ln


Hodgkin-Huxley Experiment• Voltage-clamping


H-H Voltage Clamp Experiment


HH Model: Refractory Period (1)


HH Model: Refractory Period (2)


Integrate-and-Fire Neuron• Leaky integrator

• Firing threshold

• Reset voltage

• Refractory period

j i

jij

m

ttwtI

tRItudt

tdu

)()(

)()()(

)( jitu

restj

ij utu

)(lim0


Response of IF Neurons• Membrane potential with a constant current before

spikes

mte

RIuRItu /)0(11)(


I-O Transfer Function

RIuRIt

r

RIuRIttt

or

tttotherwise

tttthenttifRIu

RIt

restmref

restmrefii

refii

iiirefi

restmi

ln

1

ln

ln

1

1

1


Noise models of IF-neurons• Noise sources

– Stochastic threshold– Random reset– Noisy integration (noisy input)


Neural Coding

j

i


Population Neuron Model


)(1

kk

m

jkjkjk

vy

bxwv

Activation Function

0if00 if1

k

kk v

vy

)exp(11)(

kk av

v


)T/vexp()v(P

-P(v)P(v)

x

11

1 ty probabili with0)(or 1 ty probabili with1

Determinisitic Neuron

Stochastic Neuron

Network Architecture

Fully-connected Feed-forward


Layered Feed-forward Convolutive

Learning Algorithms • While computers just conduct programmed

instructions, human brain learns from environments.– Supervised Learning

• Learn from teachers– Unsupervised Learning

• Learn from environments without teacher• Active (Self-searching for what need be learnt)• Need objective(s)

– Reinforcement Learning• Critique


Summary• “The genetic code was cracked in the mid-20th century. The

neural code will be cracked in the mid-21st century.”– Intelligence to Machine!

Freedom to Mankind!• Neural code may be learnt by learning from data.

– Unsupervised learning– Supervised learning– Reinforcement learning

• Need define– Neuron model– Network architecture– Learning algorithm


Documents

EE538-01.pdf