Jimmy Abbas
Neuromorphic Design of Smart Prosthetic and
Therapeutic Systems
Complex visual fields
Control of locomotion
Hippocampus - path planning dynamics at a fixed point?
Olfaction
Reward systems
Risk aversion ?
Auditory cues
Attention
Social behavior
Color
Path planning
System state performance
Sensorimotor integration
Opportunities for Neuromorphic Engineers in Medical Rehabilitation
• smart prostheses• artificial limbs• electrical stimulation systems
Smart ProstheticsNovember 9-11, 2006 - Irvine, CA
Goals:
• Enhance the climate for conducting interdisciplinary research and break down related institutional and systemic barriers
• Stimulate new modes of scientific inquiry
• Encourage communication between scientists and public
NAKFI - Themes to date:2008 - Complexity (Nov. 13-15, 2008) 2007 - Aging / Longevity (Nov. 15-18, 2007)
2006 - Smart Prosthetics2005 - Genomics2004 - Designing Nanostructures2003 - Signaling
http://www.keckfutures.org/
Medtronic Corp.
Cochlear Corp.
Alfred Mann Institute,
Cleveland FES Center
Victhom Human Bionics
Motion Control Corp.
Cyberkinetics, Inc.
Prostheses of Today and Tomorrow
Prostheses of the Future
Intelligent
Responsive
Biomimetic
Mechanical Prostheses:Artificial limbsHeart valvesCerebral shunts
Neural prostheses:Hand graspStanding/steppingBladder/bowel controlExerciseMemory/cognitionNeuromotor therapyVisionHearing
Limb lossSpinal cord injuryStrokeBrain injuryParkinson’s DiseaseDeafnessBlindnessMemory impairmentsALSCerebral palsy
Why?
What?
Task Group Topics• Describe a framework for replacing damaged cortical tissue and
fostering circuit integration to restore neurological function.• Build a prosthesis that will grow with a child (such as a heart valve or
cerebral shunt, or a self healing prosthesis).• Develop a smart prosthetic that can learn better and/or faster.• Brain interfacing with materials: recording and stimulation
electrodes.• Refine technologies to create active orthotic devices.• Structural tissue interfaces: enabling and enhancing continual
maintenance and adaptation to mechanical and biologic factors.• Sensory restoration of perception of limb movement and contact.• Design a functional tissue prosthesis.• Create hybrid prostheses that exploit activity-dependent processes.• Can brain control guide or refine limb control?
Task Group 3: Develop a smart prosthesis that can learn better and/or faster.
client prosthesis
facilitator
Task Group 3: Develop a smart prosthesis that can learn better and/or faster.
client prosthesis
facilitatorclient:• representations used in the brain• how to transmit information• mechanisms of plasticity• how to maximally exploit plasticity
Knowledge/technology gaps:
Task Group 3: Develop a smart prosthesis that can learn better and/or faster.
client prosthesis
facilitator
facilitator:• identification of intermediate milestones
(performance and neurophysiological)• individuality of minimal detectable differences• customizing for specific user groups• engage the user
Knowledge/technology gaps:
Task Group 3: Develop a smart prosthetic that can learn better and/or faster.
client prosthesis
facilitator
Knowledge/technology gaps:
prostheses:• biomorphic sensors/actuators for
compatibility w/ neural reps• detecting and communicating user
intent and motor commands• maintain/improve performance
based on dynamic interface• real-time machine learning • redundancy for versatility, efficiency
seamless integration
where are the ‘smarts’?
cooperative learning
co-adaptationmaximizing plasticity
versatility
automated fitting
minimal cognitive demandsease-of-use
neuromorphic systemsbiomimicry
coordination
sensorimotor integration
information transmission/coding
utilizing plasticity autonomous - direct control
self-repairing
high throughput, high fidelity
amazondotcomificationredundancy
Smart ProstheticsNovember 9-11, 2006 - Irvine, CA
Top five activities (from a list of 34) amputees would like to be able to perform with their electric prostheses
1) Type/use a word processor2) Open a door with a knob3) Tie shoelaces4) Use a spoon or fork5) Drink from a glass.
Transradial Users
1) Type/use a word processor2) Cut meat3) Tie shoelaces4) Open a door with a knob5) Use a spoon or fork
Transhumeral Users
Atkins et al., 1996
Surface EMG/ EEG
Implanted EMG
Direct peripheral nerve interface
Electrocorticogram
Recording w/ penetrating spinal/brain electrodes
Non-Invasive
Centrally Invasive
Command Input Source to Prosthesis
Sensory Feedback to User
Visual
Auditory
Cutaneous Stimulation
Peripheral Nerve Stimulation
Recording w/ penetrating spinal/brain electrodes
Non-Invasive
Centrally Invasive
If everyone is thinking alike then someone isn't thinking. - General George S. Patton
Peripheral Nerve
Horch; Utah, ASU
Electrocorticogram
Moran & Leuthardt; Wash. U.Williams; U. Wisconsin
Nicolelis; Duke U.
Donoghue; Brown U.Cyberkinetics, Inc.
Intracortical arrays
Muscle re-innervation
Kuiken; RIC
Where to put the ‘smarts’in smart prostheses?
Endogenoussystem:
Exogenous system:
neural control
neuromorphic control
biological sensors
engineered sensors
muscles/ tendons
biomorphic actuators
Integrating Engineered and Physiological Systems
• biomimicry• physiological engineered• adaptive systems• system integration
Oscar Pistorius
Hugh Herr, MIT
Spinal cord
Control of locomotion
Spinal cord injury
Locomotor retraining
The spinal cord is a conduit for information transfer from the brain.
The spinal cord is a highly sophisticated computational structure that mediates a wide range of physiological functions.
Recruitment modulation
Rate coding
Co-contraction
StiffnessForce-velocity relationship
Reflexes
Contraction dynamicsFatigue
Length-tension
Biarticular muscleSecondary action of a muscle
Length
tension
Key features of neuromotor control that are either mediated by or have implications for spinal cord organization
Pattern Generator (PG)
Spinal Segmental Circuitry (SSC)
SupraspinalCenters
motoneuronoutput
PGmotorcommands
PG reflexmodulation
output
Musculoskeletal System
ascendingoscillatory
drive
descendingPG inputs
descendingmotor
commands
spinalreflexpaths
PGreflexpathways
supraspinalreflex
pathways
Neural organization for locomotor control:
• Anatomical organization
• Reflexes (feedback)
• Pattern generation
• Multi-level structure
most inputs to motoneurons from the brain are indirect (via spinal interneurons)
• spinal cord mediated reflexes
• supraspinal modulation
... ...Musclespindle Muscle
spindleExtensormuscle
Flexormuscle
Descendingpathway
Motor neurons
Ia inhibitoryinterneuron
Ia afferent
Ia afferent
...... ......Musclespindle Muscle
spindleExtensormuscle
Flexormuscle
Descendingpathway
Motor neurons
Ia inhibitoryinterneuron
Ia afferent
Ia afferent
Spinal Central Pattern Generators (CPGs)
• CPG produces basic pattern of locomotion• neuromodulators (5-HT, NE, DA, NMDA)• sensory feedback integrated into CPG operation• CPG activity is affected by training
From Barbeau et al. Brain Res. Rev. 30: 27, 1999
Brain Spinal Cord
Intracellular/extracellular recordings
Brain Spinal Cord
Intracellular/extracellular recordings
The spinal cord is a conduit for information transfer from the brain.
The spinal cord is a highly sophisticated computational structure that mediates a wide range of physiological functions.
Using spinal cord models for locomotor control
Woergoetter
Hybrid Carbon:Silicon system
• neuromorphic aVLSI circuit • real-time closed loop • 1:1 phase coupling
5 sec
2 V VR
C
E
5 sec
2 V VR
C
E
Brain Spinal Cord
Ventral RootRecordings
C
E
L L
E
C
Brain Spinal Cord
Ventral RootRecordings
Brain Spinal Cord
Ventral RootRecordingsVentral RootRecordings
C
E
L L
E
CC
E
L L
E
C
Using spinal cord models for locomotor control
Jung, et al. 2001
250,000 people with SCI in US
‘complete’ vs. ‘incomplete’
cervical - thoracic
Spinal Cord Injury (SCI)
Personal impact:• sensorimotor function• mobility and transfers• cardiovascular function• bladder/bowel• respiration• sexual function• exercise and recreation
Costs of Spinal Cord Injury:• health care costs in $billions/yr• lost productivity• reduced quality of life
Kentucky Agrability
Spinal Cord Injury
Causes of SCI:
(250,000 people with SCI in US)
motor vehicle crashes
falls
acts of violence
sports
other
Spinal Cord Injury: Clinical Outcomes from the Model Systems, Stover, DeLisa, & Whiteneck, Aspen Publishers,1995
Spinal Cord Injury
At the time of injury:• 63% are < 30 years old• 80% are employed or in
school• 90% have not gone beyond
high school• 54% are unmarried
Neurologic level of injury => degree of impairment• thoracic level lesion => paraplegia (46.2%)• cervical level lesion => tetraplegia (52.9%)
Kwon et al.; The Spine Journal 4:451-464, 2004
Primary injury due to mechanical impact
Secondary injury due to inflammation and response cascade
A window of opportunity for neuroprotective interventions
In: Neurobiology of Spinal Cord Injury, Edts.R.J. Kalb and S.M. Strittmatter, Humana Press, Totowa, N.J.
rat thoracic injury
human cervical injury
Intrinsic properties of motoneurons change after SCI
From Bennett et al. J. Neurophysiol. 86: 1955, 2001
Motoneurons atrophy after spinal injury; exercise preserves structure
From Gazula et al. J. Comp. Neurol. 476: 130, 2004
... ...Musclespindle Muscle
spindleExtensormuscle
Flexormuscle
Descendingpathway
Motor neurons
Ia inhibitoryinterneuron
Ia afferent
Ia afferent
...... ......Musclespindle Muscle
spindleExtensormuscle
Flexormuscle
Descendingpathway
Motor neurons
Ia inhibitoryinterneuron
Ia afferent
Ia afferent
SCI affects reflex modulation spasticity
Bladder-sphincter dyssynergybladder hyperreflexia
Yoshimura, Prog. Neurobiol. 1999
www.robomedica.com
Promoting Locomotor Recovery after Incomplete SCI
www.litegait.com
Partial Weight Bearing Therapy (PWBT)
If everyone is thinking alike then someone isn't thinking. - General George S. Patton
Robot-assistedColumbo; (Lokomat)
IntraspinalMicrostimulation
Mushahwar, Horch, Prochazka
Epidural Spinal Cord Stimulation
Herman, HeDimitrievic
Therapist-assistedHarkema
Reflex stimulationField-Fote
Promoting Functional Recovery After SCI
• overhead harness reduces load• treadmill assists with movement• cyclic loading pattern• exploits activity-dependent plasticity
Supplement with Neuromuscular Electrical Stimulationmore complete sensory pattern to spinal cord?
increased (or more functional) plasticity?
Treadmill Training
Neuromuscular Stimulation
Neuromorphic Adaptive Control
surfaceintramuscularintraneuralcuffepimysial
Case Western Reserve Univ.
Neuromuscular Stimulation: electrode types
NeuroTECH
U. Utah
Muscle nonlinearities are exacerbated by recruitment properties and fatigue
• limited input range• steep-slope regions• changes with fatigue
0
10
20
30
40
50
60
70
0 20 40 60 80 100
stimulation level (%)
angl
e (d
eg) pre-trial
after 12 min(300 cycles)
Muscles SkeletonPatternGenerator
adaptation
PatternShaper
feedback
PG/PS Control System
PG/PS control: PG generates cyclic pattern
- structure based on neural modelPS fine tunes the pattern
- single layer adaptive neural networkObjectives:
customize stimulationaccount for nonlinearities, dynamics, etc.adjust for fatigue
d V (t)/dt = (I (V , t) + I (V , t) + I - V (t) / R ) / Ci i i i i i im syn m PM m inj m m m
)(t)-(-i VVm+1
1 = (t)yoimii
2e
i ij ij isyn j syn syn mI (t) = y (t) g (E - V (t))∑
i i iPM NMDA KCaI (t) = I (t)+ I (t)
Membrane dynamics:
Output function (firing rate):
Membrane currents:
PG Neuron Equations
i i i i iNMDA NMDA NMDA i NMDA mI (t) = K g p (t)(E - V (t))
(t)p CVE A - (t))p-(1 CEV A=dt
(t)pdi
))/-((
i
))/-((i imir
iirmi
i
βαβα ee
i i i i iKCa KCa NMDA KCa mI (t) = g [Ca (t)](E - V (t))
](t)Ca[-(t))V-E(K(t)p=dt
](t)Cad[iNMDANMDAmCaNMDANMDANMDAi
iNMDAiiiii
δρ
Burst initiation:
PG Pacemaker Currents
i i iPM NMDA KCaI (t) = I (t)+ I (t)
Burst termination:
Adapted from Brodin, 1991
Es2 Is2 Ia2
I a1 I s1 Es1
Burst initiationMutual inhibitionReciprocal inhibition
Es Exciter of synergistsIs Inhibitor of synergistsIa Inhibitor of antagonists
Inhibitory synapseExcitatory synapse
___
__
PG Circuit
actualdesired
flexorsextensors
time (sec)
angle (deg)
torque(Nm)
PS z(t)
PG y(t)
actualdesired
time (sec)
angle (deg)
torque(Nm)
PS z(t)
PG y(t)
flexorsExtensorsR1 neuron
Reflexes with phase-resetting
Phase-dependent reflexes
actualdesired
flexorsExtensorsR1 neuron
time (sec)
angle (deg)
torque(Nm)
PS z(t)
PG y(t)
Reflexes without phase-resetting
time of disturbance application (% cycle)
Percent reduction
Peak muscle torque
RMS muscle torque
Phase-dependent reflexes
PG model provides:basic movement rhythmphase-resettingphase-dependent reflexescycle period adjustments
Needs:localized learningmore complex movement patternsadditional reflexes
Muscles SkeletonPatternGenerator
adaptation
PatternShaper
feedback
PG/PS Control System
PG/PS control: PG generates cyclic pattern
- structure based on neural modelPS fine tunes the pattern
- single layer adaptive neural networkObjectives:
customize stimulationaccount for nonlinearities, dynamics, etc.adjust for fatigue
PS Neuron Equations
∑=
=m
jtkjytkjwtzk
1)()()(
muscle/skeleton
PG
PS
zy
jx Y
Ydese
Σ
Σ
0 0.2 0.4 0.6 0.8time (sec)
individual PS neuron activity levels, y1j
0
0.5
1
PS Learning
∑ −=Δn
nTtkjyn
tetkjw )(1)()( η
)()()( tytytw prepostη=Δ
)()()( tytytw preaη=Δ
Hebbian learning algorithm:
Heterosynaptic Hebbian learning algorithm:
Time-averaged learning algorithm:
system output (degrees)
0 0.2 0.4 0.6 0.8 1-0.2-0.1
00.10.2weighted PS
neuron activity levels, w1jy1j
0 5 10 15 20-0.2neuron index, j
20 40 60 80 1000
5
10
15RMS error per cycle (degrees)
0 0.2 0.4 0.6 0.8 1-20-10
01020
0 0.2 0.4 0.6 0.8 1-0.4-0.2
00.20.4
PS output, z1(t)
20 40 60 80 1000
5
10
150 0.2 0.4 0.6 0.8 1
-20-10
01020
0 0.2 0.4 0.6 0.8 1-0.4-0.2
00.20.4
0.2
0 0.2 0.4 0.6 0.8 1-0.2-0.1
00.1
0 5 10 15 20-0.2neuron index, j
time (sec) time (sec)
time (sec)
time (sec)
time (sec)
time (sec)
time (sec)
time (sec)
c5%=14 cycles c5%=4 cycles
RMS10=.57° RMS10=.50°
-0.10
0.10.2
PS neuron weights, w1j
0.2
-0.10
0.1
0 0.2 0.4 0.6 0.8 1time (sec)
0 0.2 0.4 0.6 0.8 1time (sec)
individual PS neuron activity levels, y1j
n=10, na=5, η=.002 n=10, na=15, η=.002
0
0.5
1
n(k)
0
0.5
1
Angle (deg)
0 5 10
0
20
40
Time (sec)
90 95 100
0
20
40
Time (sec)
PG/PS Desired
0 10 20 30 40 500
20
40
60
Cycle #
RMS (%)
N=32
PG/PS PD +/-STD
PG/PS Control
Adapting to account for fatigue
0 50 100 150 200 250 3000
10
20
30
40RMS error (%)
OFFOFF OFFON ON ON
Cycle #
Adaptation:
Mean ± 1 STD (n=16)
Thigh Segment
(Hip Angle)(Deg) -15
0
15
-10
10
30Shank
SegmentAngle (Deg)
0102030
Knee JointAngle (Deg)
Fixed Stim01
01
0Stim by PGPS 0
11
47 48 49 50Cycle Number
-15
0
15
-10
10
30
0102030
0 1 2 3Cycle Number
actual
desired
FWR
gluts
first 3 cycles last 3 cycles
quadshams
control of multi-segment movements
-20
-10
0
10
20First 10 cycles
Hip angle (Deg)
-20
-10
0
10
20Last 10 cycles
20 40 60 80 1000
0.2
0.4
0.6
0.8
1
% of cycle
Normalized hams stim
20 40 60 80 1000
0.2
0.4
0.6
0.8
1
% of cycle0 0
20 40 60 80 1000 20 40 60 80 1000
BF
IL
VL
ST
TA
GM
HumanPre-clinical
80
100
120 Hip
60
90
120 Knee
40
85
130 Ankle
40
65
90 Shoulder
80
115
150 Elbow
Fore Limb
1 1.5 2 2.5 3
Foot Fall
time (sec)
H
G
(deg
) F
(deg
)
E
(deg
)
D
(deg
)
C
(deg
)
B
A Hind Limb
HLLFLLHLRFLR
Pre-clinical model development (Jung, et al., ASU)
BFIL
VL
ST
TA
GM
Thota et al. J. Neurotrauma 22(4): 442-465, 2005
Can we control the stepping movement?
Can we promote functional plasticity?
CK200: FES Exercise In Clinic or Home
These projects are being supported by the NIH-NCMRR: 1R43-HD050006-01, 5R44-HD041820-04 to customKYnetics, Inc.
Disclosure: J. Abbas is part-owner and co-founder of customKYnetics, Inc.
Commercial partnership: Commercial partnership: customKYneticscustomKYnetics, Inc. , Inc.
Adaptive Stimulator for Exercise and Rehabilitation
Stimulation-Augmented Exercise and Neuromotor Therapy
Integrating physiological and artificial systems
Endogenoussystem:
Exogenous system:
neural control
neuromorphic control
biological sensors
engineered sensors
muscles/ tendons
biomorphic actuators
• biomimicry
• intact artificial
• system integration
Center for Adaptive Neural Systems
neural prostheses and advanced prosthetic systems
adaptation in neural systems
neuromorphiccontrol systems
biological inspiration
technology to replace lost or missing functionality
technology to promote learning or adaptation
neuro-inspired system design
Jimmy Abbas
Neuromorphic Design of Smart Prosthetic and Therapeutic Systems
FULTONs c h o o l o f e n g i n e e r i n g s c h o o l o f e n g i n e e r i n g
Ira A.
Acknowledgements:NIHNSFFDAUS ArmyASUBarrow Neurological InstituteBanner Good Samaritan Medical Center
JoAnne RiessPankaj KatariaXia ZhangEric HartmanJunli OuJason Gillette, PhDEd Stites, MD, PhDSusan McDowell, MD
Manoshi BhowmikAndrea DowningAlison SitekRanu Jung, PhDNarayanan Krishnamurthi, PhDKen Horch, PhDDevin Jindrich, PhDSharon Crook, PhDBertan Bakkaloglu, PhDBrian Smith, PhDSteve Philips, PhDRichard Herman, MDJohann Samanta, MDPadma Mahant, MDDenise Campagnolo, MDDan Lieberman, MD