CSH, May 23, 2005 1

Deconstructing the Law of Effect

C. R. Gallistel

Rutgers Center for Cognitive Science

CSH, May 23, 2005 2

Matching Is

• An innate strategy foraging strategy (or, possibly, an innate principle of choice = melioration)

• The apportionment of foraging time (and effort) in accord with the incomes obtained from the foraging locations: T1/T2=I1/I2, where Ti = time invested in Location i &

Ii = income obtained from Location i

Equivalently: T1/(T1+T2) = R1/(R1+R2) & T2/(T1+T2) = R2/(R1+R2) [“Herrnstein fractions”]

• An evolutionarily stable strategy

CSH, May 23, 2005 3

Income vs Return

• Income (Ii) = (reward from Loc i)/(unit time spent foraging, without regard to where time is invested, that is, summed over i)

• Return (Ri) = (reward from Loc i)/(unit time spent foraging at a given Loc i), i.e., per unit time invested in a location

• Matching equates returns;T1/T2=I1/I2 ---> I1/T1 = T2/I2

CSH, May 23, 2005 4

Model 1: Law of Effect

• Matching driven by inequality of returns

• Subject compares returns and reapportions investments to favor location yielding greater return

• If income is only loosely connected to investment, then this decreases return on favored side and increases it on disfavored side

• Eventually, a dynamic equilibrium is attained (the equation of returns)

• No change-detector necessary

• Hill climbing = slow!

CSH, May 23, 2005 5

Model 2: Innate Strategy Pure Feed Forward

• First proposed in a general way by Gene Heyman (“matching is unconditioned behavior”)

• Leaving rate parameters the inverse of the proportions among the incomes

• A separate process detects changes in income• Income estimator works on small samples taken

from the period between the time when a change point is (retrospectively) estimated to have occurred and the time at which the change is detected

CSH, May 23, 2005 6

Old Rat Results

CSH, May 23, 2005 7

Appearance of Matching in the Experimentally Naïve Mouse

• Mouse shuttles back and forth between two feeding hoppers

• Hopper entries monitored by IR beams

CSH, May 23, 2005 8

Coupling between investment and income

• Slightly coupled: traditional concurrent VIs with indefinite hold for harvesting

• Income completely independent of investment (pellets accumulate at random rates; the mouse harvests the accumulation whenever it comes

• Complete coupling: the schedule clock only for a hopper only runs when the mouse has its head in that hopper

CSH, May 23, 2005 9

Tracking Matching

• Compute difference in “Herrnstein fractions” after each feeding R1/(R1+R2)- R2/(R1+R2)=(R1- R2)/ (R1+R2)

= Feeding Imbalance; possible values -1 & 1,depending on whether feeding was at 1 or 2

T1/(T1+T2)-T2/(T1+T2)=(T1- T2)/(T1+T2)= Investment Preference; ranges -1 to 1, depending on how mouse spent its time between feedings

Cumulate differences: slope of cumulative record gives average values; when slopes of the 2 cumulative records are the same mouse is matching

CSH, May 23, 2005 10

Condition 1

• Relative income jointly determined by programmed ratio and sampling behavior

• Naïve mouse samples rarely

• Hence initial ratio of incomes may be 1:1 no matter what the programmed ratio

• Mouse is always matching

• Changes in relative income and changes in relative investment coincide (even though income only very loosely connected to investment)

CSH, May 23, 2005 11

All Mice in Cond 1

• They are, in general, as close to matching at the beginning of training as they are at the end of it

CSH, May 23, 2005 12

Condition 2

• Income imbalances now constant from beginning (as intended)

• Mice match from the beginning

CSH, May 23, 2005 13

Condition 3

• When income is directly tied to investment, the only place in behavioral parameter space for matching is exclusive devotion to one or the other option

• Thus, on our model, preference should be (almost) all-or-none, but unstable

CSH, May 23, 2005 14

A Closer Look

• The session in which the major reversal occurred

• Nothing in the returns justifies the switch!

• The change in income does justify the switch

• This change is caused by the switch, but, in our model, the animal doesn’t know that, because it takes no account of the impact of its behavior on what it observes

CSH, May 23, 2005 15

Terminal Economic Weirdness

• Animals are oblivious to the impact of their own behavior on the rewards they receive (pace Skinner & the entire economics profession)

CSH, May 23, 2005 16

Matching & theHunter-Gatherer World

• Hunting & gathering, unlike farming, do not bring food into existence, they merely exploit what is there

• Intelligent sampling of a changeable environment is essential in hunter-gathering

• Matching maximizes income when income is environment-limited rather than investment-limited

• Matching samples the world intelligently