Temporal Dynamics of Learning The Visioncseweb.ucsd.edu/~gary/cs200/s07/slides/cottrell-week1-tdlc.pdfTemporal Dynamics of Learning The Vision Time: The missing link between brain

Temporal Dynamics of Learning The Vision


The Temporal Dynamics ofThe Temporal Dynamics ofLearning Center (TDLC):Learning Center (TDLC):

Better Learning by Advancing theBetter Learning by Advancing theScience of TimeScience of Time

Garrison W. Cottrell, PI


At each point in TIMETIME, the brain changes to best meet the changing demands

of the environment.


Time: The missing link betweenbrain and behavior

• Understanding how the brainbrain learns isimportant for understanding how studentsstudentslearn in the classroom.

• And how the brain learns depends cruciallyon timetime and timingtiming.

• Here, I will develop one example that is amodel for what we hope will be replicatedmany times by our Center.


Time matters for processing…

“say”

“stay”

Freq

uenc

y

100 ms

Time (milliseconds)

These wave forms are identicalidentical except for theartificially inserted gap! Yet we all hear a /t/ inserted…


Time matters for learning…

This baby has learned that exciting things go on to her left whenthe two tones are different -- at a 70ms “boop-beep” interval…


50

55

60

65

70

75

80

85

90

95

100

8

15

30

60

150

305

428

947

1466

1985

3023

3543

4062

Interstimulus Interval (ISI) in milliseconds

Perc

en

t C

orr

ect

Control

Language Impaired

< 40ms - Phonemes

40-350ms - Syllables

Time matters for learning…Tallal & Piercy (1973) Nature

6-8 year old children with a language impairment can’t dothis task at short delays: correlation or causation?correlation or causation?


Looks like causation:• Rapid auditory processing (RAP) ability at 66

months of agemonths of age accurately predicts languageimpairment (with 91% accuracy) at 3 years of3 years ofageage.

• Remarkably, training children to make thesedistinctions with the FastForWord program canimprove reading.

• This training is grounded in basic research onneural plasticity from animal models.

Time matters for learning…

Benasich & Tallal (2002) Behav Brain Res.


Time matters for Education…

This child is trying to detect whether two tones are both rising,rising and falling, or both falling (FastForWord).


Time matters for Education…

After considerable training, the child is able to perform the taskat a much faster rate.


Reading Improvements After Intervention

Temple et al. (2003) PNAS

Compared to non-treatment controls, suchlanguage-oriented training leads to improved readingimproved reading.

70

75

808590

95100

105

110

115120

Control

Stan

dard

Sco

reReal WordReading

Non-WordDecoding

PassageComprehension

**

**

*

Dyslexic Control Dyslexic Control Dyslexic

PrePost


ControlsFrontal

&Temporo-parietal

Frontalbut NO

Temporo-parietal

DyslexicsPre-

Intervention

fMRI Activation While Rhyming Letters

Temple et al. (2003) PNAS

Brain activityBrain activity differs between controls and dyslexics:Training alters functional activity such that it more closely resembles normal activity

IncreasedIncreasedactivity inFrontal &Temporo-parietal

DyslexicsPost-

Intervention


Time matters for learning:at this short time scale…

• Abnormalities in temporaltemporalprocessingprocessing at the 20-50 mstime scale lead to poorphonemic perception…

• Which leads to a cascade ofcascade ofdetrimental learningdetrimental learningeffectseffects on ALL longer timescales: Poor language skills.

• By concentrating onremediatingremediating the basictemporal processing deficit,we can improve higher-levelperformance.

Temporal perceptual weakness

Weak phonological representations

Reading, writing, spelling problems

Learning and academic problems

Struggling students

Oral language weakness


• 5-20 ms – Synaptic integration, spike-timing dependent

plasticity (STDP), speech

• 50-200 ms – Event and motor sequence

perception/production

• 200-500 ms – Fixation duration, attentional blinks,

saccades, attention shifts, timing of social interactions

• 0.5-2 s – Perception, motor rhythms, emotional expressions

• 2-50 s – repeated exposure and training effects, reward

signals

• minutes, hours, days, months, years: Spacing effects,

memory consolidation, analytic/holistic changes in expertise

And at ManyMany Time Scales0

100 msec

1 sec

10 sec

minutes

hours

days

months


Our Challenge

• Develop a new Science of LearningScience of LearningDynamicsDynamics that treats timetime as a crucialelement of the process.

• Integrate the study of the dynamicsdynamics oflearning across multiple time scales,brain systems, individuals, and socialsystems.

• Change educational practicepractice based onsound science.


Meeting This Challenge Requires:

• Well-organized collaborationsbetween scientists frommultiple disciplines.

• Large-scale data infrastructurethat makes data persistent andaccessible to many scientists.

• Theoretical models capable ofspanning time scales.

Cottrell

Behrmann

Sheinberg

BubCurran

Tarr

Tanaka

Schultz

neuropsychologyfMRI

modelingneurophysiologyERPs

psychophysicscomputer graphics Gauthier

Palmeri


Seems like a tall order….

How will we do it?

A new kind of Center:

The Network-of-Networks Network-of-Networks research structure


The Network-of-Networks SolutionMathematics

Physics

MachineLearning

Robotics

ComputerScience

ComputationalNeuroscience

Neuroscience

Cognitive Science

Linguistics

Neuropsychology

CognitivePsychology

DevelopmentalPsychology

Learning Theory

Education

UC San Diego

RutgersUniversity

VanderbiltUniversity

UC Berkeley

University ofColorado

The Salk Institute

QueenslandUniversity

VictoriaUniversity

BrownUniversity

Carnegie-MelonUniversity

Yale University

San Diego StateUniversity

SensoriMotor Network

Perceptual Expertise NetworkInteracting Memory Systems

Social Interaction Network

BridgeMembers


Why these four?Sensorimotor Learning mediates ourSensorimotor Learning mediates our

interactions with the worldinteractions with the world

Interacting Memory SystemsInteracting Memory Systems maintainmaintaincontinuity across timecontinuity across time

Social Interaction provides the contextSocial Interaction provides the contextfor this learning - especially in thefor this learning - especially in theclassroomclassroom

Perceptual Expertise allows us toPerceptual Expertise allows us tofluently and appropriately interactfluently and appropriately interactwith the worldwith the world


Cottrell

Behrmann

Sheinberg

BubCurran

Tarr

Tanaka

Schultz

neuropsychology

fMRImodeling neurophysiology

ERPspsychophysics

computer graphics Gauthier

Palmeri

A proof of concept:The Perceptual Expertise Network (PEN):


The Perceptual Expertise Network (PEN):

A group of 10 researchers from Psychology, Neuroscience,Neuropsychology and Computer Science

The goal of our network was to understand the learningmechanisms and representations underlying visualexpertise

Our first meetings were used to establish a commonvocabulary, and to synchronize our researchsynchronize our research around aset of common questions, rather than focus on ourtechniques


Cottrell

Behrmann

Sheinberg

Palmeri

Bub

Curran

Gauthier

Tarr

Tanaka

Schultz

Year 0

Cottrell

Behrmann

Sheinberg

Palmeri

Bub

Curran

Gauthier

Tarr

Tanaka

Schultz

Year 0

Behrmann

Palmeri

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Bub

Schultz

Gauthier

Year 1Behrmann

Palmeri

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Bub

Schultz

Gauthier

Year 1

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 2

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 2

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 3

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 3

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 4

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 4

Behrmann

Curran

Gauthier

Tarr

Cottrell

Behrmann

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Behrmann

Curran

Gauthier

Tarr

Cottrell

Behrmann

Tanaka

Sheinberg

Palmeri

Bub

Schultz

SLC


Memory

Social

S-M

PEN

Year 1

Memory

Social

S-M

PEN

Year 1

Cottrell

Behrmann

Sheinberg

Palmeri

Bub

Curran

Gauthier

Tarr

Tanaka

Schultz

Year 0

Cottrell

Behrmann

Sheinberg

Palmeri

Bub

Curran

Gauthier

Tarr

Tanaka

Schultz

Year 0

Behrmann

Palmeri

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Bub

Schultz

Gauthier

Year 1Behrmann

Palmeri

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Bub

Schultz

Gauthier

Year 1

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 2

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 2

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 3

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 3

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 4

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 4


Memory

Social

S-M

PEN

Year 1

Memory

Social

S-M

PEN

Year 1

PEN

Social

Memory

S-M

Year 2

PEN

Social

Memory

S-M

Year 2

S-M

Memory

Social

PEN

Year 3

S-M

Memory

Social

PEN

Year 3

S-M

Social

MemoryPEN

Year 4

S-M

Social

MemoryPEN

Year 4

S-M

Memory

Social

PEN

Year 5

S-M

Memory

Social

PEN

Year 5

Cottrell

Behrmann

Sheinberg

Palmeri

Bub

Curran

Gauthier

Tarr

Tanaka

Schultz

Year 0

Cottrell

Behrmann

Sheinberg

Palmeri

Bub

Curran

Gauthier

Tarr

Tanaka

Schultz

Year 0

Behrmann

Palmeri

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Bub

Schultz

Gauthier

Year 1Behrmann

Palmeri

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Bub

Schultz

Gauthier

Year 1

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 2

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 2

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 3

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 3

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 4

Palmeri

Behrmann

Curran

Tarr

Cottrell

Tanaka

Sheinberg

Palmeri

Bub

Schultz

Gauthier

Year 4


Sensorimotor Network

Network Goal

To understand the temporal dynamics ofsensory perceptual learningsensory perceptual learning and motormotorlearninglearning, from biological learning rules atsynapses to human behavior.

Precise timingPrecise timing is a critical feature ofnatural sensory stimuli, and ofmovements, and is actively learned.

We will study the synaptic mechanismssynaptic mechanisms, neural systemsneural systems, and computational strategiescomputational strategies usedby the brain to implement temporal features of sensory perceptual and motor learning.


Interacting Memory Systems Network

Network Goals

How does a learner abstract thetemporal structuretemporal structure of theenvironment ?

How does the brain arbitratecontrol across the multiple memorymultiple memorysystemssystems involved in learning?

How does the timingtiming of training andtraining andtestingtesting regimes affect the rate oflearninglearning and long-term retentionretention?

We will study brain synaptic mechanismssynaptic mechanisms, electrophysiological propertieselectrophysiological properties, computationalcomputationalstrategies, strategies, and neuroimaging neuroimaging correlatescorrelates, in order to explain the temporal features thatunderlie efficient behavioral performance.


Perceptual Expertise Network

We will take a multidisciplinary approach to studying how perceptual expertiseperceptual expertisechanges as a function of timetime and across experience.

Network GoalsTo relate the short-termshort-term dynamics of perceptionperceptionand perceptual decisionsperceptual decisions to the long-termlong-termdynamics of expert learningexpert learning, and to exploit thatunderstanding to significantly improveimprovelearning.learning.



We will study active learningactive learning and teachingteaching as well as real-timereal-time social interaction.

What are the effects of timingtiming onsocial interactions and sociallearning?

How does the brain organizebehavior in real timereal time?

What are the neural mechanismsengaged during learninglearning in a socialsocialcontextcontext?

Network Goals


The Network-of-Networks SolutionMathematics

Physics

MachineLearning

Robotics

ComputerScience

ComputationalNeuroscience

Neuroscience

Cognitive Science

Linguistics

Neuropsychology

CognitivePsychology

DevelopmentalPsychology

Learning Theory

Education

UC San Diego

RutgersUniversity

VanderbiltUniversity

UC Berkeley

University ofColorado

The Salk Institute

QueenslandUniversity

VictoriaUniversity

BrownUniversity

Carnegie-MelonUniversity

Yale University

San Diego StateUniversity

SensoriMotor Network

Perceptual Expertise NetworkInteracting Memory Systems


BridgeMembers


• Cross-cutting projects that allnetworks could work on.

• Initiatives that would cause us tosynchronize research across thecenter

• Initiatives that would answer theBig Questions

But…this wasn’t good enough:NSF wanted InitiativesInitiatives!


1. What is a SIP???

The First TDLCTDLC Initiative

Is it

a): a small drink of water?b): a Strategy for Initiative Planning?c): A way for NSF to torture us?d): A Strategic and Implementation Plan?

Actual answer: All of the above!


1. How is temporal information about the worldlearned, and how do the temporal dynamics of theworld influence learning? [INPUT]

2. How do the dynamics intrinsic to brain cells, brainsystems, and behavior influence learning?[CENTRAL PROCESSING]

3. How are the temporal structures of movement andsampling the environment learned? [OUTPUT]

4. What mechanisms determine the time course oflearning itself? What general principles explain thedynamics of learning across multiple scales anddomains? [THEORY]

The Actual TDLCTDLC Initiatives


•• Each research networkEach research network has a coherent focus; thehas a coherent focus; theinitiatives weave them together:initiatives weave them together:

•• The goal of network meetingsThe goal of network meetings is to synchronizeis to synchronizeresearch research within networkswithin networks

–– ByBy bringing together researchers from multiple domains, thebringing together researchers from multiple domains, theresearch becomes research becomes problem orientedproblem oriented rather than rather than techniquetechniqueorientedoriented..

–– At At PENPEN’’s s early meetings, we designed experiments that couldearly meetings, we designed experiments that couldbebe runrun using the same stimuli in humans, monkeys, andusing the same stimuli in humans, monkeys, andneural nets: we neural nets: we synchronized our research!synchronized our research!

•• Alternating with network meetings, the goal ofAlternating with network meetings, the goal ofinitiative meetings is to synchronize research initiative meetings is to synchronize research betweenbetweennetworks.networks.

Initiative/Network threading


Where will we be in Five Years?Out of funding?


Where will we be in Five Years?Deliverables

•• A Science of the Temporal Dynamics ofA Science of the Temporal Dynamics ofLearning.Learning.

• We need to understand the temporaldynamics of learning in order to manipulateit towards better learning outcomes.

• The Fast ForWord example I described here isthe prototype.


Deliverables:New Tools for Education

• With our partner Scientific LearningCorporation, use the FastForWord FastForWordplatform to translate our research intothe classroom: e.g., Fast ForMath,Fast ForMath,FastForMemoryFastForMemory……

• LetLet’’s Face It!s Face It!, a program for teachingkids with social deficits to read faces.

• RUBIRUBI, an educational robot suitable forpreschools.


Deliverables:New Tools for Research

S-M

Memory

Social

PEN

Year 5

S-M

Memory

Social

PEN

Year 5• The network ofnetworks structure:

• Scalability

• Collaboration

• Shared facilities

• Bridge postdocs

• Shared philosophy



• A Data Sharing FacilityData Sharing Facilityto provide support formanagement of jointresearch data, and a state-of-the-art Data GRIDData GRID atthe NSF-sponsored SanDiego SupercomputerCenter to allow data mining,temporal alignment ofimaging, spike data andbehavioral data frommultiple sites.

• This will become a resourcefor all Centers

Storage Resource Broker Collections at SDSC (8/2/2005)

TBs of

datastored

Number

of files(millions)

Users

withACLs

Data Grid

NSF/ITR - National Virtual Observatory 53.9 9.5 100

NSF - National Partnership for Advanced Computational Infrastructure 36.1 7.5 380

Static collections – Hayden planetarium 8.0 0.2 227

Pzone – public collections 13.0 6.7 68

NSF/NPACI - Biology and Environmental collections 40.1 0.1 67

NSF/NPACI – Joint Center for Structural Genomics 15.7 1.6 55

NSF - TeraGrid, ENZO Cosmology simulations 176.7 2.1 3,267

NIH - Biomedical Informatics Research Network 10.6 7.6 303

Digital Library

NSF/NPACI - Long Term Ecological Reserve 0.3 0.01 36

NSF/NPACI - Grid Portal 2.6 0.05 460

NIH - Alliance for Cell Signaling microarray data 0.7 0.08 21

NSF - National Science Digital Library SIO Explorer collection 2.7 1.1 27

NSF/ITR - Southern California Earthquake Center 131.0 2.7 73

Persistent Archive

NHPRC Persistent Archive Testbed (Kentucky, Ohio, Michigan, Minnesota) 0.1 0.4 28

UCSD Libraries archive 4.1 0.4 29

NARA- Research Prototype Persistent Archive 1.5 0.9 58

NSF - National Science Digital Library persistent archive 3.6 27.0 136

TOTAL 501 TB 68 million 5,335



• A Motion capture FacilityMotion capture Facility to provide state-of-the-artequipment and software for simultaneaous capture andanalysis of multiple fine-scale temporal actions andinteractions. Hand movement

Eye movementMuscle activity

Body movement

Haptics

Force

Facial Expression



• A Brain DynamicsBrain DynamicsFacilityFacility to provide state-of-the-art simultaneousEEG/fMRI analysis andhardware for fine scaletemporal and spatialresolution of brain activityto bring off the nextrevolution in brainimaging.

Δ B

OLD

Signal (%)

-4

0

+4

EEG Alpha Power vs. BOLD


Deliverables:New Tools for Outreach

•• The Science Network WebThe Science Network WebTV Channel TV Channel to broadcast“Town Meetings” betweenscientists, policy makers,parents, teachers and the publicand help promote a betterunderstanding of science in theUS and the world.

• All of the Science of LearningCenters will be able to use TheScience Network to publicizetheir science.


Deliverables:More Diverse Scholars

• UCSD’s Preuss Preuss SchoolSchool: 100%school lunch program kids, 91%accepted to four-year colleges:They are in our labs, and theyhelped to give RUBI locomotion.

•• Reach for TomorrowReach for Tomorrow: Innercity kids come to campus and areinspired to attend college

•• Faculty Partners ProgramFaculty Partners Program:Building pipelines to minorityinstitutions


Summary

• The time is right to study time!

• We have an excellent team of world-classresearchers.

• We have a well-designed and flexibleorganizational structure.

• We have a unity of vision.


Summary

• We are prepared to train a diverse group ofscientific leaders for the 21st century.

• We are committed to making our researchrelevant to the classroom.


Life passes in milliseconds,

but what we learn

in those milliseconds

changes us for life.



Managing and Evaluating Our CenterOur home will be at CalIT2

which uses a management modelbased on the Apollo Program

Thorough Multi-layeredEvaluation Plan

Andy Porter of Vanderbiltwill advise us on evaluation:And he advised us to use Brenda Turnbull!

Our External Advisory Board will evaluate our scientific progress

Documents

Temporal Dynamics of Learning The Visioncseweb.ucsd.edu/~gary/cs200/s07/slides/cottrell-week1-tdlc.pdfTemporal Dynamics of Learning The Vision Time: The missing link between brain