14
Efficient Neuromorphic Computing with the Feedforward Inhibitory Motif 1 Yu (Kevin) Cao, 2 Maxim Bazhenov, 1 Jae-sun Seo, 1 Shimeng Yu, 3 Visar Berisha 1 School of ECEE, ASU; 2 School of Medicine, UCSD; 3 Dept. of SHS, ASU

Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Embed Size (px)

Citation preview

Page 1: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Efficient Neuromorphic Computing with

the Feedforward Inhibitory Motif

1Yu (Kevin) Cao, 2Maxim Bazhenov, 1Jae-sun Seo, 1Shimeng Yu, 3Visar Berisha 1School of ECEE, ASU; 2School of Medicine, UCSD;

3Dept. of SHS, ASU

Page 2: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

A top-down approach: better for CPU/GPU– Pros: mathematical, accurate, scalable– Cons: computation cost, energy efficiency, off-line learning

IBM Jeopardy2880 3.5GHz

P7 cores

Google Cat:16,000 CPU

cores

Machine Learning Today

- 2 -

Edge computing needs novel hardware/algorithms– Local to the sensor, real-time, reliable, low-power– On-line, personalized learning with continuous data

30 frames/s

Page 3: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Acceleration Needs 103 – 105 speedup required to achieve real-time training

of HD images at 30 frames/second

- 3 -

GPU10 – 30 X

FPGA10 – 50 X

ASIC102 – 103 X

Beyond CMOS >103 X

Ij

ViRij

Ij = Σ(1/Rij)⋅Vi

Resistive Crossbar Architecture

Page 4: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Physical Challenges Nonlinear, noisy, poor endurance (habituation in programming)

- 4 -

A bio-plausible hardware-algorithm solution: robust, low-power, low-precision, accurate, on-line

These hardware problems (variations, unreliable synapse) and application demands (real time, on-line learning, and mobile) exist in biological cortical and sensory systems!

10k 20k 30k 40k 50k 60k

40

50

60

70

80

90

Realistic (with resistivesynaptic device)

Ideal (software)

0 200 400 600 800 100020

30

40

50

60

70

80

90

100

∆C

ondu

ctan

ce (%

)

Number of Write

Decay in RRAM Write

(Habituation)

Page 5: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Brain-inspired Computing A bottom-up approach: better integration with sensors

– Pros: energy efficiency, simpler computing, real time, reliable– Cons: complicated dynamics, limited scale and accuracy

- 5 -

Neuron4-100μm[22nm]

MicrocircuitFO = 1K-100K

[FO = 4]

System100B, 100Hz, 20W, 30% ER/neuron

[1.4B, 3.7GHz, 45W, <10-9 BER]

Task Complexity (log)

Mac

hine

Com

plex

ity (L

og)

CPU

Brain

Neural Computer

Page 6: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Neurobiological Basis of Learning

Reward (supervision): global feedback signal

Inhibition: unsupervised sparse feature extraction

Synapse: non-linear, habituation (local), noisy

Neurons: continuous leaky-integrate-fire

Learning: local, feed forward STDP or SRDP on each plastic synapse

- 6 -

Monkey, Parietal cortex, Nature Communications, 2015

Insect, olfactory system, Nature Neuroscience, 2007

Mouse, Motor cortex, Nature Communications, 2014

Page 7: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

RHINO: A Biomimetic Solution Reward, Habituation, Inhibition, NOise Motif: a recurring network element; general in biological process

- 7 -

LHIs

KCs

AL

+ ‒

+

Mushroom Body (MB)

Antennal Lobe (AL)

Kenyon Cells (KCs) 15,000

Lateral Horn Interneurons (LHIs), 100

[Nature Review, 2007]

Page 8: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Network Structure Rewarding for associative

(supervised) learning Inhibition to speed up the

formation of sparsity Habituation (decay in

learning rate) to achieve the convergence

Non-gradient based: no backward propagation

Local adaptation: no crosstalk among synapses

- 8 -

Input (X)

Output (E)

Inhibition (I)

Classifier (C)

Reward

Page 9: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Learning Rules Reward: A global feedback to all W’s

– C: classification score– Correct: |C-Cth|; Punish: -|C-Cth|; If |C-Cth| < θC, no reward feedback

Classification: Punish only – ΔW∝ -(C-Cth)E/η for wrong classification

Excitation: Hebbian learning rule with habituation– ΔW∝ Reward⋅E⋅(X- θXE)/η

– η: learning rate decays with training, i.e., habituation per synapse

Inhibition: positive feedback on E– If E < θweak, ΔW∝ Reward⋅E⋅I/η

– If E > θstrong, ΔW∝ Reward⋅(E-ρ)⋅I/η

Neuron: spiking leaky-integrate-fire

- 9 -

0 ρθstrongθweak

InhibitionPlasticity

∆ W

E

Page 10: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Demonstration: MNIST MNIST for handwriting recognition

– Data represented by 0 – 50 spikes– Full image 28 x 28– No pooling or normalization– 50% connectivity of WX2E and WX2I

- 10 -

E: 2000

C: 10

X: 28 x 28

I: 100

0k 10k 20k 30k 40k 50k 60k4

6

8

10

12

14

16

without inhibition with inhibition

0 20 40 60

82

84

86

88

90

92

94

96

RHINO Sparse coding No feedforward I

Page 11: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Neuron Firing Rate Sparsity: an appropriate range (5-15%) is critical Homeostatic balance, which controls overfiring of the output

neurons, is essential for learning

- 11 -

Firin

g R

ate

of 2

000

E N

euro

ns

Handwriting Digits (10 categories)0 1 2 3 4 5 6 7 8 9

With homeostatic balance

Without homeostatic balance

0 1 2 3 4 5 6 7 8 90 1 2 3 4 5 6 7 8 9

Beforetraining

Page 12: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

w/ inhibition (E + I) w/o inhibition (E only)

[Microsoft, 2015]

LHIs

KCs

AL

+ ‒

+

Factors for Learning Accuracy Initial randomness: without noise, learning cannot start With 100 Is, the network size of E is reduced by 3X at the same

accuracy of 95%; ~50X speedup over gradient-based approaches

- 12 -

0K 2K 4K 6K 8K

75

80

85

90

100% 40% 80% 20% 60% 10%

[Microsoft, 2015]

Page 13: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Results Comparison

- 13 -

Reference Input Data format and precision

Learning rules

Number of neurons

Number of parameters

Number of images Accuracy

Mushroom body 28x28 Spike Rewarded STDP 50000 5E5 60000 87%

Two layer SNN 28x28 Spike STDP 300 2.4E5 60000x3 93.5%

Unsupervised SNN 28x28 Spike STDP 6400 4.6E7 200000 95.0%

This work 28x28 Spike rate in a 50 window

RewardedSRDP 2100 8.4E5 60000 95.0%

This work 28x28 Spike rate in a 50 window

Rewarded SRDP 6000 2.4E6 60000 96.2%

Spiking RBM 28x28 Spike rate Contrastive divergence 500 3.9E5 20000 92.6%

Sparse Coding 10x10 patch 3-bit number Gradient 300 3E4 60000x10 94.0%

Two layer NN 28x28 Floating number

Gradient descent 1000 7.8E5 60000 95.5%

Spiking CNN 28x28 Spike timing Regenerative learning 5.6E4 1.2E5 60000 99.08%

Page 14: Efficient Neuromorphic Computing with the Feedforward ...ornlcda.github.io/neuromorphic2016/presentations/Yu-Cao-RHINO-v2.pdf · Efficient Neuromorphic Computing with the Feedforward

Summary RHINO: A bio-plausible spiking NN

– Feedforward inhibitory motif– Reward + Local adaptation

What matters to efficiency: spiking, precision, motif,…?

- 14 -

Algorithms– Multi-layer, hierarchical – Low-precision learning– On-line learning

Hardware– Implementation with

resistive synaptic array – Reliable learning

E1: 500

E2: 1500

X: 28x28

X1: 28x28