Valuation and choice: Using values

PSYC 596F10/20/2009

Presenter: David Yokum

Chapter 28: Antonio Rangel. The computation and comparison of value in goal-directed choice.

Chapter 29: Michael Platt and Camillo Padoa-Schioppa. Neuronal representations of value.

Outline

• Rangel: How does the brain compute and compare values? The random-walk model.

• Platt and Padoa-Schioppa: What are the neural substrates of value computations? OFC, LIP, and CGp.

• Throughout, your questions and comments!

Value-based decision making

1) Create representation of decision problem2) Assign values to considered actions3) Compare valuations

(select highest valued action)4) Measure desirability of outcomes5) Use feedback to update

Simplifications1) Create representation of decision problem

2) Assign values to considered actions3) Compare valuations

(select highest valued action)4) Measure desirability of outcomes5) Use feedback to update

Goal-directed choice:– two options– familiar outcomes– immediate consumption

Random dot motion task

Fig. 28.1

Random-walk model (RWM): measure of relative evidence

1. Encode moment-to-moment sensory evidence2. Accumulator measuring net evidence3. Thresholds implementing decision criterion

Fig. 28.2

RWM: behavioral evidence

Fig. 28.3

RWM: neural correlates (1)

• Middle temporal area contains groups of neurons that respond preferentially to stimuli moving in certain directions (i.e., response fields); activity increases with coherence.

Fig. 28.5

RWM: neural correlates (2)• Time-courses of neural activity in lateral

intraparietal area rise to single threshold about 100 ms before saccade; an accumulator signal?

Fig. 28.6

RWM: neural correlates (3)

• What about threshold computation?• Maybe LIP? Maybe superior colliculus?

Simple goal-directed choice

Fig. 28.7

RWM redux: attention-driven choice

• Relative value again computed, but this time an advantage is given to whatever item is being attended to.

Fig. 28.8

AD-RWM: behavioral evidence

Fig. 28.9

Random walk model: Evaluation?

• Filippo: Rangel is worried that the model is limited to binary choice, but this concern is unwarranted.

• How is threshold determined?

Neural substrates of value computations

• Orbitofrontal cortex (OFC): absolute value of good

• Lateral intraparietal parietal cortex (LIP): relative value of good

• Posterior cingulate cortex (CGp): subjective value; update LIP

Economic choice task

Fig. 29.1

Economic choice task

Fig. 29.1

OFC and chosen value (i.e., value of whatever juice ultimately chosen)

Fig. 29.2

OFC and offer value (i.e., value of one of the two juices alone)

Fig. 29.3

OFC and taste

Fig. 29.3

OFC invariant for menu changes

Fig. 29.4

OFC upshot

• Representation of value in OFC is independent of visuo-motor contingencies and menu invariant.

• Essentially a cardinal, or absolute, representation of value.

OFC data: Evaluation?

• Trevor: “I don't know much about how dementia works. When dementia patients have OFC damage, is it selective to a particular part of the OFC, or to neurons in the OFC that either get their inputs from or project to specific regions, or is it just widespread, unselective damage to the OFC? If it is selective, is it possible that the cell death corresponds to neurons of particular functional types (chosen, offer, or taste) neurons?”

OFC data: Evaluation?

• Trevor: “The authors present at least 4 different functional types [of info encoded by OFC neurons]: chosen value, offer value A, offer value B, and taste. Do we know anything about the properties of these different neurons? Are "offer" neurons and "chosen" neurons different in any meaningful ways or part of different functional networks in the brain?”

OFC data: Cardinal versus ordinal• Filippo: Problems with conclusion that OFC activity

represents cardinal value. Opposing data from Tremblay and Schultz (1999). “[M.P. and C.P-S.] briefly discuss the clash between the two lines of research by noting that i) they may have recorded from different orbital areas and ii) that the paradigm adopted in Tremblay et al. may have induced adaptation, which in turn may explain the differences between the two sets of results. They then conclude this brief line of reasoning by claiming that the “cardinal hypothesis” is confirmed by empirical data.

“Apart from the fact that the strength of the argument is questionable, it is also not clear how one can discern encoding of ‘absolute value’ from other possible information…”

• Luke: “…the conclusion that it [OFC] calculates "objective" value seems overstated and tenuous at best.”

Lateral intraparietal cortex (LIP)

• Linking sensory signals with motor commands, as well as guiding sensory attention.

• Modulated by expected value of different stimuli?– Enter Michael Platt’s c.v. – LIP neuronal activity proportional to EV of target

(e.g., altered juice amount or probability of cue)

LIP: pay-per-view task

Fig. 29.5

Fig. 29.6

LIP: abstract, relative value

LIP signal independent of modality

• “These findings suggest that decisions based on value operate on a common currency that is independent of the modality of the goods under consideration or the actions they motivate” (450).

Question

• Phil: “Is there a difference between "primary" rewards (such as food/drink), "secondary" rewards (such as money), and "social" rewards (such as viewing pictures of attractive members of the opposite sex): Clearly the value of food/drink varies with how hungry the animal (rat/monkey/human) is, while a secondary reward has value only in that it can be exchanged for a primary reward (i.e. Monopoly money is fairly useless), but social rewards might be different - for instance, can one ever get bored of looking at photos of pretty women? If hunger/thirst mediate the value of food/drink, what might mediate the value of photos of attractive women?...”

Question

• Trevor: “I found the brief discussion of mutlimodality valuation very interesting and wanted more information about studies in humans that have tried to assess whether valuation is done the same across different types of information. Aside from the studies with female and male images discussed in the chapter, and some of the charitable giving research, are there any good examples of studies that have tried to look at decisions in which multiple modalities must be considered when making a choice?”

LIP data: Evaluation?

• Mirre: “Are LIP neurons only involved in perceptual decision-making or has LIP activity only been measured using saccadic decision-making tasks? Do LIP neurons in human beings function similarly as in monkeys?”

LIP data: Evaluation? • Kaisa: “The fact that different economical variables are reflected in the LIP has

been puzzling me for quite a while. What does it really tell us in studies where monkeys are answering with eye movements? Somehow I have gotten the feeling that always when there is a significant difference in the behavior, LIP neurons also show the same effect (sometimes amazingly accurately). Are the measurements of LIP activity and behavioral responses (saccades) actually indicating the same thing at different stages, i.e., selection of motor action and the action itself? Is the choice calculated elsewhere and fed forward to LIP? Could the monkeys still detect the direction of random dot motion if they would not have means of responding at the time of motion display (no knowledge which saccade to make)? If the monkeys could learn to use delayed response instructions (appearing after the motion display disappeared), what kind of activity would there be in LIP prior to the saccade (fast accumulation in every condition)? Does anybody know if this kind of studies have been made?

And if monkeys are able to make 'abstract' choices about the direction of motion, which areas are involved in the computations? There could be a region coding for the choice irrespective of any spatial location, similarly to the OFC coding value of goods.”

LIP data: Evaluation?

• Luke: “No disrespect to Michael Platt and colleagues, but I found something lacking in their argument and overall approach. For reasons that have never been clearly spelled out, he has spent over a decade studying value in regions in which there is no clear evidence (nor ever has been) supporting their role in its calculation. At the conclusion of the chapter he notes what has been painfully obvious to everyone else for years - lesioning the LIP does not result in an impaired ability to calculate value. However, it does impair attention and ability to initiate visual saccades (a behavior that is crucial for his tasks…”

“…Not determined to give up on studying regions that have no clear role in calculating value, he decides to focus on the posterior cingulate gyrus…” - Luke

Posterior Cingulate Cortex (CGp)

• Anatomical and neurophysiological evidence suggest a role in “signaling motivationally significant events and actions, as well as perhaps their subjective value for guiding future behavior.”

CGp signals are subjective

CGp and evaluation of stimuli

• CGp neurons respond most strongly to visual events that are unpredictable in space or time (Dean et al., 2004)

• CGp neurons respond following delivery of unpredictable reward or omission of predictable reward (McCoy et al., 2003)

• Attention for learning

CGp: Evaluation?

• Luke: “Not determined to give up on studying regions that have no clear role in calculating value, he decides to focus on the posterior cingulate gyrus. However, rather than actually describing how it might be involved in calculating value, he gives us a list of random citations justifying how it might play a role (sounding strangely like a grant proposal). After reading a dense 20 page chapter on value, I was hoping to have at least learned something about how value might be calculated in the brain, but instead I felt like I was reading a 20 page justification of why we should pick random areas in the brain that are not well studied and then not be surprised when nothing conclusive emerges.

CGp: Evaluation?

• Luke: “Why are goal values (i.e. motivations) always neglected in models of value? If things like appetitive desires (presumably processed in the hypothalamus) are crucial for calculating goal values, why in the world would we think the furthest region away from them (both spatially and evolutionarily) would be an interesting place to begin? How might internal motivational states (e.g. hunger, sexual drive, temperature regulation) be integrated with external demands (e.g. reputation)?”

Question

• Assuming neuronal correlates of value exist in the OFC, LIP, and CGp, why are there so many representations of value in the brain?

• MP & C P-S: value signals expressed in different areas contribute to different processes– Frontal areas: economic choice– Sensory areas: perceptual choice– Motor areas: motor choice

Question

• Mirre: “Different neural correlates of value have been found in the brain. In sensory areas, value signals contribute to perceptual attention, in frontal areas, value signals contribute to economic choice and in motor areas value signals contribute to action selection. How are these signals integrated? Which value signal determines the eventual choice? What happens when there is a conflict between value signals?”

Goods- versus Actions-based

• GBM: economic choice is modular and before action selection

• ABM: economic choice “unfolds as a process of action selection”

• For GBM, choice should be “completely processed within an abstract representation of goods” (e.g., in the OFC)

Other questions

• Trevor: “…Both chapters this week (and last week) sort of handwaved at the notion that understanding the valuation process would be beneficial for clinical research, but never really justified this claim. How do you think understanding how valuation is accomplished in the brain can inform assessment and treatment of various clinical disorders?”

Other questions • Luke “Things that are more interesting to discuss then the scientific career of

Michael Platt might include:

Is there a unitary valuation system? I think this will prove to be a hotly contested issue. Are abstract representations somehow funneled into the OFC or ventral striatum and then converted into a unitary currency? Or are there independent systems involved in risk (insula, Nacc) reward (NAcc, amygdala, OFC), and punishment (insula, amygdala)? What about more abstract representations like future consequences? What about motivational values originating either exogenously or endogenously?

Is there any reason to believe that the assumptions about choice held by economists have any validity in the brain? Is there actually a system involved in representing value and another one for selecting the action with the highest expected value?