View
223
Download
5
Category
Tags:
Preview:
Citation preview
CogRIC Workshop
Adaptive Working Memory:From Computational Neuroscience
ModelTo Robot Control Module
David C. NoelleAssistant Professor of Computer Science
Assistant Professor of PsychologyVanderbilt University
david.noelle@vanderbilt.edu
August 17, 2006
CogRIC Workshop
Adaptive Working Memory:From Computational Neuroscience
ModelTo Robot Control Module
David C. NoelleAssistant Professor of Computer ScienceAssistant Professor of Cognitive Science
University of California, Merced
david.noelle@vanderbilt.edu
August 17, 2006
CogRIC Workshop
Adaptive Working Memory:From Computational Neuroscience
ModelTo Robot Control Module
David C. NoelleAssistant Professor of Computer ScienceAssistant Professor of Cognitive Science
University of California, Merced
david.noelle@vanderbilt.edu
August 17, 2006
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Adaptive Working Memory Project
Funded by NSF ITR program (EIA-0325641)www.cecs.missouri.edu/~skubic/WM/
Joshua Phillips
Kaz KawamuraMitch WilkesMarge SkubicJim Keller
Julia HighWill Dodd
Palis RatanaswasdMert Tugcu
Sam BlisardBob Luke
Stephen Gordon
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Julia HighWill Dodd
Palis RatanaswasdMert Tugcu
Sam BlisardBob Luke
Stephen Gordon
Adaptive Working Memory Project
Funded by NSF ITR program (EIA-0325641)www.cecs.missouri.edu/~skubic/WM/
Joshua Phillips
Kaz KawamuraMitch WilkesMarge SkubicJim Keller
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Working Memory
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Working Memory
Working memory systems are those thatactively maintain transient information that is
critical for successful decision-makingin the current context.
A working memory system can be viewed as arelatively small cache of
task relevant information that isstrategically positioned to
efficiently influence behavior.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Working Memory In The Brain
● A number of brain regions are implicated as important components of the human working memory system.
● One important region is dorsolateral portions of prefrontal cortex.
● Working memory is exhibited in delay period activity.
● Cells have been found which encode for locations, visual features, and association rules.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Modeling Collaborators
Todd Braver
Jonathan Cohen
Randy O'Reilly
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Active Maintenance● How are high neural firing rates sustained over a
delay?
● Mutual excitation of neurons.
● Dense recurrent connections inprefrontal cortex. Stripe sets.
● Attractor network computational models.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Controlling PFC UpdatingHow does PFC know when to actively maintain its current working memory contents? How does it know when to abandon a given working memory “chunk” in favor of a new one? The dynamics of recurrent attractor networks are insufficient to meet the simultaneous constraints of (1) active maintenance in the face of distraction and (2) rapid updating when needed. A dynamic gating mechanism is needed.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Mesolimbic Dopamine (DA) System
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Dopamine (DA) Cells
(Schultz, Apicella, and Ljungberg, 1993)
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Temporal Difference Learning● DA neurons seem to encode for change in expected
reward.
● This is equivalent to the key variable, called temporal difference error, in a powerful reinforcement learning algorithm called temporal difference (TD) learning.
● The brain may learn to produce rewarding motor sequences using a neural implementation of TD learning (Montague, Dayan, and Sejnowski, 1996).
● There are extensive DA projections to PFC. If TD learning is used to learn when to produce a given overt action, perhaps it can be used to learn when to produce a covert action – like updating working memory (Braver and Cohen, 2000).
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Computational Cognitive Neuroscience Models
● Healthy performance on frontal tasks.
● Prolonged frontal developmental period.
● Monkey lesion data.
● Human frontal damage patient performance.
● Autistic peformance.
(Rougier, Noelle, Braver, Cohen, and O'Reilly, 2005)
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Robotic Working Memory
● The highly limited capacity of working memory, along with its tight coupling with deliberation mechanisms, might alleviate the need for costly memory searches.
● Information needed to fluently perform the current task is temporarily kept “handy” in the working memory store.
Could robot control systems benefit from the inclusion of a working memory system?
Can computational neuroscience models of the working memory mechanisms of the human brain shed light on the design of a robotic working memory system?
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Potential Uses● Focus attention on the most relevant features of the
current task.
● Guide perceptual processes by limited the perceptual search space.
● Provide a focused short-term memory to prevent the robot from being confused by occlusions.
● Provide robust operation in the presence of distracting irrelevant events.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Adaptive Working Memory
● Hand Coding – For relatively routine and well understood tasks, designers may hand code procedures for the identification of useful chunks.
● Learning – If the robot is expected to flexibly respond in novel task situations, or even acquire new tasks, it would be beneficial to have a means to learn when to store a particular chunk in working memory.
How does the working memory system know when a given chunk of information should be actively maintained in working memory?
The central focus of this project is on assessing the utility of adaptive working memory mechanisms for robot control.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
The Working Memory Toolkit● Memory traces, or chunks, are pointers to arbitrary
C++ data structures.
● The adaptive working memory toolkit (WMtk) requires the user to specify:
the capacity of the working memory
a function which extracts features from chunks
a function which provides relevant features of the current system state
a function which provides instantaneous external reward information
● The toolkit provides a function for examining the contents of working memory, returning chunk pointers.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
On Each Time Step ...● The robot control system making use of the WMtk
suggests candidate chunks for retention by the working memory.
● A component of the TD learning algorithm, called the adaptive critic, is used to estimate the expected future reward of retaining various combinations of chunks. The collection of chunks with the highest expected future reward value are remembered (with high probability).
● The amount of instantaneous external reward received on this time step, along with the estimates of the adaptive critic on this time step and on the previous time step, are used to compute the TD error – the change in expected future reward. This error signal is used to train the adaptive critic.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Delayed Saccade Task
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Robotic Delayed Saccade Task
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Task Structure● Three kinds of “goal” working memory chunks:
Stay fixated on the object that you are looking at.
Look at the last location of the crosshair.
Look at the last location of the target.
● The robot control system obeys any goal chunks in working memory (resolving “look at” conflicts at random). If no chunks are being actively maintained, the system looks at a randomly selected object or, when there are no displayed objects, at a random location on the screen.
● Note that remembering all chunks will often lead to failure.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Results
The robot successfully learns the task ...
... after about 4000 trials.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Revisiting The Neuroscience● Dopamine cells sometimes fire in a way that does
not reflect a change in expected future reward. Specifically, they often fire in novel situations.
● If the dopamine signal is seen as TD error, this suggests that brain treats novel situations as if they were more predictive of reward than is warranted by experience.
● This has been implemented in the WMtk through the incorporation of an optimistic critic – the TD algorithm is initialized to predict high future reward for novel combinations of chunks.
● With this modification ...
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Optimistic Critic Results
The robot successfully learns the task ...
... after about 300 trials.
An improvement by more than a factor of ten!
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Other Preliminary Successes● Mitch Wilkes and his students have used the WMtk
to allow a physical mobile robot to ...
... learn which percepts to approach in order to produce the largest amount of forward motion down a hallway.
... learn which percepts reliably identify a specific target location, where reward is received.
● Marge Skubic and her students have used the WMtk to allow a simulated robot to learn which motor program to retain as a goal chunk, given the current sensory state, so as to solve a water maze problem.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Summary● Computational cognitive neuroscience models of the
interactions between the prefrontal cortex and the midbrain dopamine system have been successful at accounting for a variety of working memory phenomena.
● The basic structure of these models, involving the use of a reinforcement learning algorithm to learn, from experience, what should be retained in working memory and what can be safely forgotten, has been abstracted into an open source software library called the Working Memory Toolkit.
● By attending to nuances of biology, the adaptive learning capabilities of the toolkit have been greatly improved.
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Questions?
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
The End
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
Extra Slides
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
PFC Stripes & Thalamic Loops
David C. Noelle, Ph.D.david.noelle@vanderbilt.edu
people.vanderbilt.edu/~david.noelle
A More Complex Network
Recommended