Seeing Patterns in Randomness:

Irrational Superstition or

Adaptive Behavior?

Angela J. Yu

University of California, San Diego

March 9, 2010

“Irrational” Probabilistic Reasoning in Humans

1 2 2 2 2 2 1 1 2 1 2 1 …1 2 2 2 2 2

Random stimulus sequence:

1 2 1 2

• “hot hand”

• 2AFC: sequential effects (rep/alt)

(Gillovich, Vallon, & Tversky, 1985)

(Soetens, Boer, & Hueting, 1985)

(Wilke & Barrett, 2009)

“Superstitious” Predictions

Subjects are “superstitious” when viewing randomized stimuli

O o o o o o O O o O o O O…

repetitions alternations

slow slowfast fast

Trials

• Subjects slower & more error-prone when local pattern is violated

• Patterns are by chance, not predictive of next stimulus

• Such “superstitious” behavior is apparently sub-optimal

“Graded” Superstition

(Cho et al, 2002)(Soetens et al, 1985)

[o o O O O]RARR = or [O O o o o]

RT

ER

Hypothesis:

Sequential adjustments may be

adaptive for changing environments.

tt-1t-2t-3

Outline

• “Ideal predictor” in a fixed vs. changing world

• Exponential forgetting normative and descriptive

• Optimal Bayes or exponential filter?

• Neural implementation of prediction/learning

I. Fixed Belief Model (FBM)

A (0)R (1) R (1)

hiddenbias

observedstimuli

?

…

?

II. Dynamic Belief Model (DBM)

A (0)R (1) R (1)

changingbias

observedstimuli

?

?

.3 .8.3

QuickTime™ and a decompressor

are needed to see this picture.

RA bias

What the FBM subject should believe about the bias of the coin,given a sequence of observations: R R A R R R

FBM Subject’s Response to Random Inputs

FBM Subject’s Response to Random Inputs

What the FBM subject should believe about the bias of the coin,given a long sequence of observations: R R A R A A R A A R A…

QuickTime™ and a decompressor

are needed to see this picture.

RA bias

What the DBM subject should believe about the bias of the coin,given a long sequence of observations: R R A R A A R A A R A…

QuickTime™ and a decompressor

are needed to see this picture.

RA bias

DBM Subject’s Response to Random Inputs

Randomized Stimuli: FBM > DBM

Given a sequence of truly random data ( = .5) …

FBM: belief distrib. over

Simulated trials

Pro

bab

ility

DBM: belief distrib. over

Simulated trials

Pro

bab

ility

Driven by long-term average Driven by transient patterns

“Natural Environment”: DBM > FBM

In a changing world, where undergoes un-signaled changes …

FBM: posterior over

Simulated trials

Pro

bab

ility

Adapt poorly to changes Adapt rapidly to changes

DBM: posterior over

Simulated trials

Pro

bab

ility

Persistence of Sequential Effects

• Sequential effects persist in data

• DBM produces R/A asymmetry

• Subjects=DBM (changing world)

FBM

P(s

tim

ulus

)

DBM

P(s

tim

ulus

)

Human Data(data from Cho et al, 2002)

RT

Outline

• “Ideal predictor” in a fixed vs. changing world

• Exponential forgetting normative and descriptive

• Optimal Bayes or exponential filter?

• Neural implementation of prediction/learning

Bayesian Computations in Neurons?

Optimal PredictionWhat subjects need to compute

Too hard to represent, too hard to compute!

Generative ModelWhat subjects need to know

(Sugrue, Corrado, & Newsome, 2004)

Simpler Alternative for Neural Computation?

Inspiration: exponential forgetting in tracking true changes

Exponential Forgetting in Behavior

Exponential discounting is a good descriptive model

Linear regression:R/A R/A

Human Data

Trials into the Past

Coe

ffic

ien

ts

(re-analysis of Cho et al)

Linear regression:R/A R/A

Exponential discounting is a good normative model

DBM Prediction

Trials into the Past

Coe

ffic

ien

ts

Exponential Forgetting Approximates DBM

Discount Rate vs. Assumed Rate of Change

…

DBM

= .95

Simulated trials

Pro

bab

ility

= .77

Simulated trials

Trials into the Past

DBM Simulation

Coe

ffic

ien

ts

Human Data

Trials into the Past

Coe

ffic

ien

ts

= .57 = .57

Reverse-engineering Subjects’ Assumptions

= p(t=t-1)

= .57 = .77

changes once every four trials

2/3

Analytical Approximation

Quality of approximation vs.

.57

.77

nonlinear Bayesian computations 3-param model

1-param linear model

Outline

• “Ideal predictor” in a fixed vs. changing world

• Exponential forgetting normative and descriptive

• Optimal Bayes or exponential filter?

• Neural implementation of prediction/learning

Subjects’ RT vs. Model Stimulus Probability

Repetition Trials

R A R R R R …

Subjects’ RT vs. Model Stimulus Probability

Repetition Trials

R A R R R R …

RT

Subjects’ RT vs. Model Stimulus Probability

Repetition Trials Alternation Trials

R A R R R R …

RT

Subjects’ RT vs. Model Stimulus Probability

Repetition vs. Alternation Trials

Multiple-Timescale Interactions

Optimal discrimination(Wald, 1947) 2

1

• discrete time, SPRT

• continuous-time, DDM

DBM

(Yu, NIPS 2007)

(Frazier & Yu, NIPS 2008)(Gold & Shadlen, Neuron 2002)

SPRT/DDM & Linear Effect of Prior on RT

Timesteps

RT hist

Bias: P(s1)

<RT>

Bias: P(s1) x

tanh x

0

SPRT/DDM & Linear Effect of Prior on RT

Empirical RT vs. Stim Probability

Bias: P(s1)

<RT>

Predicted RT vs. Stim Probability

Outline

• “Ideal predictor” in a fixed vs. changing world

• Exponential forgetting normative and descriptive

• Optimal Bayes or exponential filter?

• Neural implementation of prediction/learning

Neural Implementation of Prediction

Leaky-integrating neuron:

• Perceptual decision-making(Grice, 1972; Smith, 1995; Cook & Maunsell, 2002; Busmeyer & Townsend, 1993; McClelland, 1993; Bogacz et al, 2006; Yu, 2007; …)

• Trial-to-trial interactions(Kim & Myung, 1995; Dayan & Yu, 2003; Simen, Cohen & Holmes, 2006; Mozer, Kinoshita, & Shettel, 2007; …)

bias input recurrent

=1/2 (1-) 1/3 2/3

Neuromodulation & Dynamic Filters

Leaky-integrating neuron:

bias input recurrent

Norepinephrine (NE)

(Hasselmo, Wyble, & Wallenstein 1996; Kobayashi, 2000)

Trials

NE: Unexpected Uncertainty

(Yu & Dayan, Neuron, 2000)

Learning the Value of Humans (Behrens et al, 2007) and rats (Gallistel & Latham, 1999)

may encode meta-changes in the rate of change,

Bayesian Learning

00 1

.3 .9.3

…

…

…

…

Iteratively compute joint posterior

Marginal posterior over

Marginal posterior over

• Neurons don’t need to represent probabilities explicitly

• Just need to estimate

• Stochastic gradient descent (-rule)

Neural Parameter Learning?

learning rate error gradient

€

ˆ α n ← ˆ α n−1 + ε(xn − ˆ P t ) ˆ P t′

€

Pt′ = − 1

6 (1− β )−2 + 13 Qt−1

€

Qt−1 = x t−1 + βQt−2 + 2Pt−1 − 1−α1−β

€

Q1 = x1

Learning Results

Trials

Stochastic Gradient Descent

Trials

Bayesian Learning

Summary

H: “Superstition” reflects adaptation to changing world

Exponential “memory” near-optimal & fits behavior; linear RT

Neurobiology: leaky integration, stochastic -rule, neuromodulation

Random sequence and changing biases hard to distinguish

Questions: multiple outcomes? Explicit versus implicit prediction?

Unlearning Temporal Correlation is Slow

Marginal posterior over

Marginal posterior over

Trials

Pro

bab

ilit

yP

rob

abil

ity

(see Bialek, 2005)

Insight from Brain’s “Mistakes”

Ex: visual illusions

(Adelson, 1995)

(Adelson, 1995)

lightness

depth

context

Neural computation specialized for natural problems

Ex: visual illusions

Insight from Brain’s “Mistakes”

Discount Rate vs. Assumed Rate of ChangeIterative form of linear exponential

Exact inference is non-linear

Linear approximation

Empirical distribution

Bayesian Inference

Posterior

Generative Model(what subject “knows”)

1: repetition

0: alternation

Optimal Prediction(Bayes’ Rule)

Bayesian Inference

Optimal Prediction(Bayes’ Rule)

Generative Model(what subject “knows”)

Power-Law Decay of MemoryHuman memory

Stationary process!

Hierarchical Chinese Restaurant Process

10 7 4 …

(Teh, 2006)

Natural (language) statistics

(Anderson & Schooler, 1991)

Ties Across Time, Space, and Modality

Sequentialeffects

RT

Stroop

GREENSSHSS

Eriksen

time

modalityspace

(Yu, Dayan, Cohen, JEP: HPP 2008)

(Liu, Yu, & Holmes, Neur Comp 2008)

Sequential Effects Perceptual Discrimination

Optimal discrimination(Wald, 1947) R

A

• discrete time, SPRT

• continuous-time, DDM

DBM

PFC

(Yu & Dayan, NIPS 2005)

(Yu, NIPS 2007)

(Frazier & Yu, NIPS 2008)(Gold & Glimcher, Neuron 2002)

Monkey G

Coe

ffic

ien

ts

Trials into past

= .72

Exponential Discounting for Changing Rewards

Monkey F

Coe

ffic

ien

ts

Trials into past

= .63

(Sugrue, Corrado, & Newsome, 2004)

Monkey G

Coe

ffic

ien

ts

Trials into past

= .72

Monkey F

Coe

ffic

ien

ts

Trials into past

= .63

Human & Monkey Share Assumptions?

MonkeyHuman

≈!

= .68 = .80

Simulation Results

Trials

Learning via stochastic -rule

Monkeys’ Discount Rates in Choice Task(Sugrue, Corrado, & Newsome, 2004)

Monkey FC

oeff

icie

nts

Trials into past

= .63

.63

.68

Monkey G

Coe

ffic

ien

ts

Trials into past

= .72

.72

.80

Human & Monkey Share Assumptions?

.72

.80

.63

.68

MonkeyHuman

≈!