1

Functional neuroimaging of emotion

Patrik Vuilleumier

Laboratoire de Neurologie du Comportement et Neuro-Imagerie Cognitive,

Dept de Neuroscience – CMU& Dept de Neurologie - HUG,

Université de Genève

http://labnic.unige.ch

E = mc2

???

Another attempt (1881)

“[In Mosso’s experiments] the subject to be observed lay on a delicately balanced table which could tip downward either at the head or at the foot if the weight of either end were increased. The moment emotional or intellectual activity began in the subject, down went the balance at the head-end, in consequence of the redistribution of blood in his system.”

-- William James, Principles of Psychology (1890)

Angelo MossoItalian physiologist

(1846-1910)

Brain imaging today

Anatomical MRI

Diffusion MRI

Functional MRI

EEG / MEG

PET/SPECT

2

IRMf (BOLD contrast = blood oxygen level dependent)

Neuroimagerie fonctionnelle

Comment fonctionne l’IRMf?

• Champ magnétique: 1.5 - 3.0 Tesla - jusqu'à 7 T c/o homme, 14 T c/o animal- champ terrestre = 50 microT

• champ signal & résolution spatiale

IRM structurelle / anatomique

(3 Tesla, résolution sub-millimétrique)

3

IRMf = fonctionnelle

Temps

Stimuli, tâche, etc.

EPI, images T2*

Effet « BOLD »(Blood Oxygen Level Dependent contrast)

• images T2*

• effet paramagnétique deoxy Hb > oxy Hb

• contraste “endogène” (Segawa et al. 1990-91)

État de base (repos) Activation neuronale

IRMf du cortex visuel

V1 area

V5 / MT area

StimulationRVF LVF

4

Régions corticales spécialiséespour la perception des visages

Un réseau spécialisé:gyrus occipital (OFA)gyrus fusiform (FFA)sillon temporal supérieur (STS)amygdale…

Perception des visages chez le singe macaque:IRMf + électrophysiologie

Tsao et al., Science 2006

IRMf:foyers d’activation sélective

enregistrement neuronal:

302/310 (= 97%) des neurones sont sélectifs aux visages

Logothetis et al., 2001

Réponse BOLDcorrèle avec champspotentiels locaux (LFP)

inputs synaptiques?

IRMf et enregistrement neuronal simultané

Visual cortex

Electrode

5

To address questions about…

• brain architecture

• individual differences

• psychiatric disorders

• art and music

• morality and philosophy

• Freud and consciousness

Using imaging to explore emotions

A "recent" field in (neuro) sciences

Advertisement !

6

“Everyone knows what an emotion is, until asked togive a definition. Then, it seems, no one knows.”

Fehr & Russell (1984) J. Exp. Psychol. 113: 464-486

What is an emotion ?

An adaptive response of the organism to some events (in some context), with multiple components:

• motor

• autonomic• perception / attention• memory• subjective “feeling”

Aversive Pleasant

Aroused

Apathetic

How many emotions ?

Anger

Disgust

Fear

Surprise

Sadness

Neutral

Joy

• 6 basic categories (Ekman 1969)• 2 basic dimensions (Russel 1980)• 4 basic neural circuits (Panksepp 1982)

Anger

Disgust

Fear

Surprise

Sadness

Neutral

Joy

Aroused

Aversive Pleasant

Apathetic

Emotional modules?

7

The emotional brain

OrbitofrontalAnt cingulateStriatumAmygdalaSensory I & IIInsula

(+ other subcortical)

Amygdala and fear

16 msec

- amygdala typically associated with fear & conditioning (but not only)

- responds to fear expression in faces (but also to others)

- activates even without awareness (though it can be modulated)

- orchestrates a wide range of behavioral and autonomic responses

Based on J. LeDoux 1996)

magno-cellular vs parvo-cellular visual pathways

= +

8

Amygdala vs Cortex: differents visual inputs

Amygdala

Nature Neuroscience 2003, 6(6), 624-631

lowSF

Fusiform gyrus

highSF

+

+

Low SF High SF

=

=

“Hybrids”

9

LB

CE

AB

OF cx

hippocampus

amygdala

V1

FFAITC

Direct feedback influences from amygdalaon sensory cortical areas

(e.g. Amaral et al., 1992)

Visual cx

Autonomicoutput

med lat

Neutral facesHouses

0 5 10 15-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

% s

ign

al f

MR

I

Time post-stimulus onset (sec)

The "Fusiform Face Area” : enhanced activation to fear

Fearful faces

Neuron 2001:30:829-841

FFA

How quickly our brain perceives important events

10

Visual search: faster when « odd » face is emotional

Average detection time = 2020 ms Average detection time = 1650 ms

N= 18, p<.001

Sabatinelli et al., 2005

Amygdala activation drives visual responses to various stimuli

Emotionally arousing > Neutral Pictures

Visual search in phobia

11

Patients withintact amygdala(n= 13)

Faces > Houses Emotional > Neutral

Nature Neuroscience 2004, 7:1271-8

no modulation of FFA by emotionwhen amygdala sclerosis

Patients withamygdalasclerosis(n= 13)

Distant consequences of amygdala lesions(medial temporal lobe sclerosis)

Conclusion

• Perceptual processing in sensory areas is guided by emotion signals from amygadala

• Individual biases related to specific fears (phobia)

Functional impact on a distributed network

Hypothalamus

Visual cx (FFA, pSTS, V1)

Retrosplenial cx

Rostral cingulate & vmPFC

Right somatosensory>Parametric SPMof amyg sclerosis severity (T2 FLAIR)on response to fear > neutral faces

HippocampusNat Neurosci 2004, 7:1271-8

Please note!

12

L rostral cingulate / vmPFC

Differential responses to fearful faces

N= 12

Anxious inhibition (BIS score)

fMR

I re

spon

se

Anxiety (STAI-sate)20 30 40 50 60

-1.5

-1

-0.5

0

0.5

1

1.5

fMR

I re

spon

se

Role of individual personality traits

(p <.001)

Differential responses to angry voices

N= 15

R rostral cingulate / vmPFC

(p <.001)

Neuroimage 2005, 28(4):848-58

VMPFC, emotion regulation, and depression

Drevets et al., Nature 1997 (PET) Chronicdeep brain stimulation

H. Mayberg et al. Neuron 2005

(ll) (sl/ss)

Genetic polymorphism in healthy people: example of 5HT LTR

Hariri et al. Science 2002

Negative correlation

Positive correlation

Pezawas et al. Nature Neuro 2005Caspi et al., Science 2003

13

« endophenotype(s) » risk factorsor « bio-marker(s) »

Psychiatricillness

Environment

Genes

Imaging neural circuits mediating between gene and behavior

Cells Systems Behavior

Treatments

Negative correlation RRS

Whole-brain multiple regression

including rumination, depression, and anxiety scores

Resting, parietal PCA map

Visual cortex

Visual task, switch vs repeat

Visual task, repeat vs switch

Insula

Ongoing depressive ruminations

Positive correlation RRS

Resting, cuneus PCA map

Enthorhinal cortex

Visual task, repeat vs switch

Conclusions

• Some psychopathological traits correlate with responses in brain areas involved in the regulation of emotions (e.g. vmPFC, sgACC)

• Changes in regional brain activity may provide new « biomarkers » (endophenotypes) for some psychiatricconditions or symptoms

• Facilitate translational research (circuit models in animals)

14

Real-time fMRI and neuro-feedback

Weiskopf et al., J Physiol 2004

• Learning to control pain sensation:Participants can modulate activity in their own anterior cingulate cortex (ACC), which is seen “online”

De Charms et al., PNAS 2005

***

* **

• After learning: decrease of reported pain intensity for same stimulus decrease of evoked activity in ACC

Real-time fMRI and neuro-feedback

Beyond BOLD amplitude:multivoxel pattern analysis

15

Brain “reading” and pattern classification

Haynes et al., NN 2006 Kamitani et al., NN 2006Kay et al., Nature 2008

Accurate classification of new image: 92% correct out of 120, 82% out of 1000

(n= 1750)

Pattern classification analysis in FFA

?=

?=

- e.g. support vector machines, etc…

Faces VoicesBodies

Banse & Scherer, 1996 Atkinson et al., 2004 Belin et al., 2003

AngerFearDisgustSadnessJoy

vs Neutral

X

FFA pSTS EBA TVA

n=16

Emotions can be expressed in multiple modalities

all modulated by basic emotionsselective for stimulus category

16

vs ?

Emotion-specific patterns in “voice area”

Curr Biol 2009; 19(12):1028-33

Multi-voxel pattern analysis (MVPA)

and classification

Faces VoicesBodies

Banse & Scherer, 1996 Atkinson et al., 2004 Belin et al., 2003

AngerFearDisgustSadnessJoy

vs Neutral

X

Emotions can be expressed in multiple modalities

n=16

AmygdalaMPFC

all emotions > neutralin all modalities

Can we predict emotion in one modalitybased on pattern of response to another?

Supra-modal representations

Ventromedial prefrontal cx(vmPFC)

Correlation with same emotion in other modalityvs. correlation with other emotions

?

17

Supra-modal representations

Ventromedial prefrontal cx(vmPFC)

But no difference in the amygdala for correlations within vs. between emotions:

- insufficient resolution? - overlapping representations?

Correlation with same emotion in other modalityvs. correlation with other emotions

?

*

*

(*) average across F-V, F-B, and V-B pairs: p < .001

Supra-modal representations

right MPFC

left pSTS

But no difference in the amygdala for correlations within vs. between emotions:

- insufficient resolution? - overlapping representations?

Pairwise correlations and confusions

(across the 3 modalities)

r values

18

Conclusions

• It is possible to « decode » emotion category fromdistribution of cortical activity in modality-specificregions (e.g. faces, voices)

• Supra-modal representations of emotion exist in MPFC, partly reflecting valence but not arousal dimensions

• Their exact nature remains to be clarified (emotion specific or more general role in mental states attributions)

Beyond transient events:dynamic nature of emotions

and whole brain network approaches

Lasting impact of emotions on brain states

• Emotions are not only ¨acute¨ and transient responses to external stimuli

But…

• can also have long-lastingeffects on mental and bodily states

19

Functional network architecture identifiedthrough spontaneous fluctuations at rest

D. Zhang et al. (2010) Nature Reviews Neurology 6 , 15‐28.

- MPFC, PCC, Precuneus, IPL- rest > active task- self reflective processes, memory, interoceptive awareness and homeostasis

How is this “resting” activity perturbed by (transient) emotions?

+

restfilm

+

restfilm

Brain at (un)restafter emotional movies

+

rest

Post-FEAR<

Post-NEUTRAL

Post-JOY<

Post-NEUTRAL

T-value0

2

4

6

vMPFC

Precuneus

0

2

4

6

T-value

dACC

dMPFC

Precuneus

Neutral

Fearful

Joyful

Effects of prior emotions (induced by movies)

Neuroimage 2011, 54(3):2481-91

film

- on subsequent rest

film

film

Neutral

Fearful

Joyful

film

Effects of prior emotions (induced by movies)

++

++

++

FA FU NA NU-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

% S

igna

l cha

nge

NegativePositiveNeutral

FA FU NA NUAttended AttendedIgnored Ignored

Fearful faces Neutral faces

Movies

- on subsequent task

Faces:fearfulor neutral

left amygdala[-24, -6, -26]

R

Fearful > Neutral faces

p<.001

20

PCC-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5movies resting

Insula-2

-1.5

-1

-0.5

0

0.5

1

1.5movies resting

fearful

joyful

neutral

Precuneus-1.5

-1

-0.5

0

0.5

1

1.5restingmovies

Differential recovery of resting activity

3 successive time-bins(90 sec each)

ACC-2

-1.5

-1

-0.5

0

0.5

1

1.5movies resting

(and vmPFC)

Neuroimage 2011, 54(3):2481-91

Dynamic functional connectivity

time

0 5 10 min ROIs = 1 to n

RO

Is =

1 t

o n

BOLD fMRI

wavelet coefficient correlation(different frequency subbands)

individual ROIs (e.g. AAL atlas)

Neuroimage 2011, 56(2):616-26

Neuroimage 2011, 54(3):2481-91

Emotional effect on resting functional connectivity

• Post FEAR restingACC and PCC more connected to insula;

insula also more connected to IPL,thalamus, and pallidum.

Amygdala less coupled to VMPFC and precuneus.

• Post JOY resting

ACC and PCC more connected to IPL;

insula also more connected to IPL, thalamus, and pallidum

Decreased correlation in post-fear rest

Incr

ease

d co

rrel

atio

n in

pos

t-fe

ar r

est **

**** **** *

** *

*

** ** 0

0.5

1

1.5

2

2.5

3

3.5Ins_LIns_RACCPCC

IPL_LIPL_RPrec_LPrec_R

vMPFCAmy_LAmy_RThal_LHypoth

Pal_Rz-value

Decreased correlation in post-joy rest

Incr

ease

d co

rrel

atio

n in

pos

t-jo

y re

st

** ** *

** *

** * 0

0.5

1

1.5

2

2.5

3

3.5Ins_LIns_RACCPCC

IPL_LIPL_RPrec_LPrec_RvMPFCAmy_LAmy_RThal_LHypoth

Pal_Rz-value

• Higher subband (0.06-0.11 Hz) • Lower subband (0.03-0.06 Hz)

21

Prec

Amy

ACC vmPFC

25 30 35 40 45 50-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

STAI-T score

Nor

mal

ized

cor

rela

tion

diffe

renc

e

Precuneus - Amygdala

p< .00085

25 30 35 40 45 50-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

ACC - Amygdala

STAI-T score

Nor

mal

ized

cor

rela

tion

diff

eren

ce

p< .0051

Reduced connectivitypost FEAR > post JOY movies

individual differences in regulation processesoperating after emotional eventsare closely related to anxiety tendencies

NB: no correlation with absolute activation magnitude in single regions

Changes in functional connectivity are determined by individual anxiety levels

Shifts in amygdala reactivityare amplified by state anxiety

L R

Post positive movies Post negative movies

Conclusion

• Transient emotions modulate subsequent activity in brain networks associated with the “default mode”

- reduction in self-related thinking after emotion induction- associated with decreased activity in vMPFC (post fear > joy)

• Insula-centered network in post-fear and post-joy resting

- enhanced connectivity with ACC (post fear) and IPL (post joy), but also PCC, thalamus, and striatum- consistent with role in feeling states and emotion regulation

• Functional disturbances in same networks associated with anxiety

22

0.25-0.50 Hz 0.10-0.25 Hz 0.06-0.10 Hz 0.03-0.6 Hz

A

B

Functional connectivity patterns

• Network summary statistics

• Differences between states

• Pattern classification (e.g. SVM)

frontal

occipital

*

Different frequency sub-bands

Decoding affective statesfrom network connectivity patterns

Fear?

Neutral?

Joy?

+

rest

+

rest

Decoding affective state at rest

sample

sam

ple

transformed dissimilarity space

10 20 30 40 50 60

10

20

30

40

50

60

5

10

15

20

25

30

35

Rest post JOY

Rest post FEAR

connectivity matrices for 90 regions (computed at 0.03-0.11 Hz)

dissimilarity vector relative to neutral condition (direct embedding)

pattern classification analysis (decision tree)

4005 connections x 3 states

(joy, fear, neutral)

Dissimilaritybetween graphs

classificationmovie

movie

23

+

rest

+

rest

Decoding affective state at rest

sample

sam

ple

transformed dissimilarity space

10 20 30 40 50 60

10

20

30

40

50

60

5

10

15

20

25

30

35

Rest post JOY

Rest post FEAR

connectivity matrices for 90 regions (computed at 0.03-0.11 Hz)

dissimilarity vector relative to neutral condition (direct embedding)

pattern classification analysis (decision tree)

47/60 correctly classified(78.3%)

44/60 correctly classified(73.3%)

movie

movie

n= 15 subjects,4 epochs tested for each emotion

Differential importance of connections

Rest post FEAR vs JOYvmPFC-Amy_L

Insula_L-IPL_L Insula_L-Amy_R

vmPFC-Amy_R

vmPFC-IPL_R

WORK IN PROGRESS

Different configuration of functional networkscontaining partly similar regions

Obsessive-Compulsive Disorder (OCD)

• Chronic anxiety disorder

• Frequent: 2-3% prevalence

• Characterized by

- intrusive thoughts (obsessions) producing uneasiness, apprehension, fear, or worry; and- repetitive behaviors (compulsions) aimed at reducing anxiety.

Anomalies in spontaneous brain activity and connectivityeven “at rest”?

24

CaudateSMA

Amy

SPL

MidFC

IPL

IPL

Insula

MidFC

vMPFC

ACC

Caudate

Insula

*** *

* **

**

** ***

*** *

*

0

0.5

1

1.5

2

2.5

3

3.5Caudate_LCaudate_R

SMANuc_ACC_LNuc_ACC_R

ACCvMPFC_LdLPFC_LdLPFC_R

lOFC_LlOFC_RInsula_LInsula_R

Amy_LAmy_RSPL_LSPL_RIPL_LIPL_R

Mid_FC_LMid_FC_R

Prec_RSTG_LSTG_R

CO

NT

RO

LS

vs.

OC

D

OCD (n=16) vs CONTROLS (n=16)

Brain connectivity at (un)rest in OCD

• Enhanced connectivity of striatum and amygdala with fronto-limbic regions (VMPFC), repetitive intrusion of obsessive thoughts during resting

• Enhanced connectivity between premotor (SMA, midFC) and attention control areas (ACC, IPL) top-town regulation and inhibition of compulsive actions

Fast band (0.14-0.28 Hz)Unmedicated

Resting functional connectivity and OCD severity

6 8 10 12 14 16 18-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Obsession YBOCS score

Norm

alized

 functional connectivity

Coupling between vmPFC and left caudate (free mind wandering condition)

r = 0.55p = 0.028

CaudateSMA

Amy

SPL

MidFC

IPL

IPL

InsulaMidFC

vMPFC

ACC

Caudate

Insula

25

Brain reading exerciceTypical fMRI activation to Music vs. Control Sound condition (30s epochs):

0

2

4

6

X=27 Y=-18

X=0

X=12 Y=9

Z=-3

Ventr. striatumRetrospl. cortexHPC

rACC

Visualcortex

Motor cortex

HPC

Pleasant affective evaluation

Memory and navigation

Visual imagery

Motor imagery

a mental “ballad” ? ...

• Music can induce emotions - but not basic categories!

• Psychological models suggest a specific domain of musical or “aesthetic” emotions…

• e.g. Zentner et al. 2008: 9 typical categories

Premotorcortex

STGSTG

CaudatePulvinar

N= 15 listeners,SPMs p ≤ 0.001

Z=-3

STG

Ventr.striatum Insula

VTA

Ventr.striatum

Hippoc

Sensory cortex

Ventr.striatum

HPC

VMPFC

Left HPC

Right HPC

HPC

PHG

VMPFC

PrecuneusDMPFC

Listening to 27 musical pieces (classic), 30-40 sec each,emotional ratings on 9 affective categories after each piece,

during fMRI

Factor analysis of subjective ratings

26

Z=-18

X=-3

Right hippocampus

Subgenual ACC

TENSPOW JOY WONTRATENDPEA NOS0

0.4

0.8

1.2

Mean xyz = 24.6, -13.4, -16.6Cluster size = 21 voxels

SAD

0

0.4

0.8

1.2

Mean xyz = -2.8, 31.6, -4.2Cluster size = 41 voxels

TENSPOW JOY WONTRATENDPEA NOS SAD

Y=15

Left caudate head

Right premotor cortex

-0.6

-0.2

0

0.2

0.6Mean xyz = 6.8, 16.7, 2Cluster size = 23 voxels

-0.4

0

0.4

0.8

1.2

Mean xyz = 61.7, 1.7, 31.5Cluster size = 14 voxels

TENSPOW JOY WON TRATENDPEA NOS SAD

TENSPOW JOY WON TRATENDPEA NOS SAD

Z=30

Differentiation between emotion categories

Main A+V- Main A-V+

– Affective space of musical emotions is consistent with the two basic dimensions of valence and arousal

– But more specific components seem related to memory (hippocampus), reflective processes (vmPFC), or motor circuits

– Finer distinction within each general emotion type may reflect a relative difference in the recruitment of similar components

Conclusions

Functional brain imaging can now be used to

• delineate circuits activated by various emotions (basic or more complex)

• determine factors that modulate such activations (stimulus features, task demands or context, personality, genes, psychopathology)

• guide new models and new therapeutic interventions

• test psychological theories or address philospohycontroversies (perhaps)

In summary

27

But research on emotions is still in its infancy,much remains to be learned…

Thank you for your attention !

Rationalist Model:

Emotions in moral judgments and decisions

Social Intuitionist Model:

Jonathan Haidt (2001).Psych Review, 108, 814-834.“The emotional dog and its rational tail: A social intuitionist approach to moral judgment„

Intuition = judgment including valence

(good-bad, like-dislike), made without effortful

steps of searching, pondering, or inferring

(unlike reasoning).

28

The « Trolley Problem »Two different situations in which you can sacrifice one human life to save five

Situation 1: ok

Situation 2: not ok

Science 2001, 293: 2105-2108.

• Both are social and "self-conscious" emotions

• Distinction remain controversial in psychology andphilosophy (Teroni & Deonna, 2008)

• Guilt is linked to transgressions of norms and might rely onrepresentations of others (i.e. victim of own misbehavior) –unlike shame (Baumeister et al., 1994).

• Shame might entail a stronger self-focus and be characterizedby subjective devaluation of the self – unlike guilt (e.g. Tangney 1999)

Moral emotions: Guilt and shame

* What are the common / distinct neural substrates? * Is there any difference in self- or other-referential processing?

Moral emotions: Guilt and shame

• Autobiographical memory paradigm in the fMRI scanner

• Based on private keywords (indirect questionnaire prior to scanning)

• Participants asked to relive episodes associated with:

- Guilt- Shame- Sadness- Neutral

(but emotion was not named)

lat OFC

General role in the retrieval of emotional memories

[Guilt+Shame+Sadness] > Neutral

0

2

4

6

8

10

dmPFCvmPFC

retrosplenial cx

visual cx

temporal pole anterior insula

29

Guilt > (Shame + Sadness)

Guilt-specific activation

Be

ta E

stim

ate

s (A

rbitr

ary

Uni

ts)

-0,2

-0,1

0,0

0,1

0,2

0,3

Guilt Shame Sadness Neutral

0

1

2

3

4

R lateral OFC L dmPFC

Trait Guilt Score

1,8 2,0 2,2 2,4 2,6 2,8 3,0 3,2 3,4 3,6 3,8 4,0

Be

tas

Est

ima

tes

for

Gu

ilt (

Arb

itrar

y U

nits

)

-0,6

-0,4

-0,2

0,0

0,2

0,4

0,6

0,8

r = 0.75p = 0.001

Guilt Inventory Questionnaire (Jones et al.,2000):= individual proneness to experience guilt

(e.g. “I worry a lot about things I have done in the past”)

R OFC

No shame-specific activation (shame > others)

Cereb Cor (2011)

Sociopathy and the historical case of Phineas Gage (1848)

Overlap with guilt

Overlap with both guilt and shame

Representations of self and other

Me

Me

Pessimistic

1  2  3  4 Me

Punctual

1  2  3  4 Me

Shy

1  2  3  4

Self > Other

…

Other

Other

Pessimistic

1  2  3  4 Other

Punctual

1  2  3  4 Other

Shy

1  2  3  4

…

Other > Self

Cereb Cor (2011)

30

Conclusion

• Both emotions recruit other-related representations (in STS, TPJ, retrosplenial cx): both are « social emotions »

• In addition, guilt also recruits self-related representations (in rostral ACC, ant insula): violations of norms as « internalized personal values »

• Guilt-specific activation in lateral OFC is unrelated to self or other: more general role in prediction of affective outcomes