1
Functional neuroimaging of emotion
Patrik Vuilleumier
Laboratoire de Neurologie du Comportement et Neuro-Imagerie Cognitive,
Dept de Neuroscience – CMU& Dept de Neurologie - HUG,
Université de Genève
http://labnic.unige.ch
E = mc2
???
Another attempt (1881)
“[In Mosso’s experiments] the subject to be observed lay on a delicately balanced table which could tip downward either at the head or at the foot if the weight of either end were increased. The moment emotional or intellectual activity began in the subject, down went the balance at the head-end, in consequence of the redistribution of blood in his system.”
-- William James, Principles of Psychology (1890)
Angelo MossoItalian physiologist
(1846-1910)
Brain imaging today
Anatomical MRI
Diffusion MRI
Functional MRI
EEG / MEG
PET/SPECT
2
IRMf (BOLD contrast = blood oxygen level dependent)
Neuroimagerie fonctionnelle
Comment fonctionne l’IRMf?
• Champ magnétique: 1.5 - 3.0 Tesla - jusqu'à 7 T c/o homme, 14 T c/o animal- champ terrestre = 50 microT
• champ signal & résolution spatiale
IRM structurelle / anatomique
(3 Tesla, résolution sub-millimétrique)
3
IRMf = fonctionnelle
Temps
Stimuli, tâche, etc.
EPI, images T2*
Effet « BOLD »(Blood Oxygen Level Dependent contrast)
• images T2*
• effet paramagnétique deoxy Hb > oxy Hb
• contraste “endogène” (Segawa et al. 1990-91)
État de base (repos) Activation neuronale
IRMf du cortex visuel
V1 area
V5 / MT area
StimulationRVF LVF
4
Régions corticales spécialiséespour la perception des visages
Un réseau spécialisé:gyrus occipital (OFA)gyrus fusiform (FFA)sillon temporal supérieur (STS)amygdale…
Perception des visages chez le singe macaque:IRMf + électrophysiologie
Tsao et al., Science 2006
IRMf:foyers d’activation sélective
enregistrement neuronal:
302/310 (= 97%) des neurones sont sélectifs aux visages
Logothetis et al., 2001
Réponse BOLDcorrèle avec champspotentiels locaux (LFP)
inputs synaptiques?
IRMf et enregistrement neuronal simultané
Visual cortex
Electrode
5
To address questions about…
• brain architecture
• individual differences
• psychiatric disorders
• art and music
• morality and philosophy
• Freud and consciousness
Using imaging to explore emotions
A "recent" field in (neuro) sciences
Advertisement !
6
“Everyone knows what an emotion is, until asked togive a definition. Then, it seems, no one knows.”
Fehr & Russell (1984) J. Exp. Psychol. 113: 464-486
What is an emotion ?
An adaptive response of the organism to some events (in some context), with multiple components:
• motor
• autonomic• perception / attention• memory• subjective “feeling”
Aversive Pleasant
Aroused
Apathetic
How many emotions ?
Anger
Disgust
Fear
Surprise
Sadness
Neutral
Joy
• 6 basic categories (Ekman 1969)• 2 basic dimensions (Russel 1980)• 4 basic neural circuits (Panksepp 1982)
Anger
Disgust
Fear
Surprise
Sadness
Neutral
Joy
Aroused
Aversive Pleasant
Apathetic
Emotional modules?
7
The emotional brain
OrbitofrontalAnt cingulateStriatumAmygdalaSensory I & IIInsula
(+ other subcortical)
Amygdala and fear
16 msec
- amygdala typically associated with fear & conditioning (but not only)
- responds to fear expression in faces (but also to others)
- activates even without awareness (though it can be modulated)
- orchestrates a wide range of behavioral and autonomic responses
Based on J. LeDoux 1996)
magno-cellular vs parvo-cellular visual pathways
= +
8
Amygdala vs Cortex: differents visual inputs
Amygdala
Nature Neuroscience 2003, 6(6), 624-631
lowSF
Fusiform gyrus
highSF
+
+
Low SF High SF
=
=
“Hybrids”
9
LB
CE
AB
OF cx
hippocampus
amygdala
V1
FFAITC
Direct feedback influences from amygdalaon sensory cortical areas
(e.g. Amaral et al., 1992)
Visual cx
Autonomicoutput
med lat
Neutral facesHouses
0 5 10 15-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
% s
ign
al f
MR
I
Time post-stimulus onset (sec)
The "Fusiform Face Area” : enhanced activation to fear
Fearful faces
Neuron 2001:30:829-841
FFA
How quickly our brain perceives important events
10
Visual search: faster when « odd » face is emotional
Average detection time = 2020 ms Average detection time = 1650 ms
N= 18, p<.001
Sabatinelli et al., 2005
Amygdala activation drives visual responses to various stimuli
Emotionally arousing > Neutral Pictures
Visual search in phobia
11
Patients withintact amygdala(n= 13)
Faces > Houses Emotional > Neutral
Nature Neuroscience 2004, 7:1271-8
no modulation of FFA by emotionwhen amygdala sclerosis
Patients withamygdalasclerosis(n= 13)
Distant consequences of amygdala lesions(medial temporal lobe sclerosis)
Conclusion
• Perceptual processing in sensory areas is guided by emotion signals from amygadala
• Individual biases related to specific fears (phobia)
Functional impact on a distributed network
Hypothalamus
Visual cx (FFA, pSTS, V1)
Retrosplenial cx
Rostral cingulate & vmPFC
Right somatosensory>Parametric SPMof amyg sclerosis severity (T2 FLAIR)on response to fear > neutral faces
HippocampusNat Neurosci 2004, 7:1271-8
Please note!
12
L rostral cingulate / vmPFC
Differential responses to fearful faces
N= 12
Anxious inhibition (BIS score)
fMR
I re
spon
se
Anxiety (STAI-sate)20 30 40 50 60
-1.5
-1
-0.5
0
0.5
1
1.5
fMR
I re
spon
se
Role of individual personality traits
(p <.001)
Differential responses to angry voices
N= 15
R rostral cingulate / vmPFC
(p <.001)
Neuroimage 2005, 28(4):848-58
VMPFC, emotion regulation, and depression
Drevets et al., Nature 1997 (PET) Chronicdeep brain stimulation
H. Mayberg et al. Neuron 2005
(ll) (sl/ss)
Genetic polymorphism in healthy people: example of 5HT LTR
Hariri et al. Science 2002
Negative correlation
Positive correlation
Pezawas et al. Nature Neuro 2005Caspi et al., Science 2003
13
« endophenotype(s) » risk factorsor « bio-marker(s) »
Psychiatricillness
Environment
Genes
Imaging neural circuits mediating between gene and behavior
Cells Systems Behavior
Treatments
Negative correlation RRS
Whole-brain multiple regression
including rumination, depression, and anxiety scores
Resting, parietal PCA map
Visual cortex
Visual task, switch vs repeat
Visual task, repeat vs switch
Insula
Ongoing depressive ruminations
Positive correlation RRS
Resting, cuneus PCA map
Enthorhinal cortex
Visual task, repeat vs switch
Conclusions
• Some psychopathological traits correlate with responses in brain areas involved in the regulation of emotions (e.g. vmPFC, sgACC)
• Changes in regional brain activity may provide new « biomarkers » (endophenotypes) for some psychiatricconditions or symptoms
• Facilitate translational research (circuit models in animals)
14
Real-time fMRI and neuro-feedback
Weiskopf et al., J Physiol 2004
• Learning to control pain sensation:Participants can modulate activity in their own anterior cingulate cortex (ACC), which is seen “online”
De Charms et al., PNAS 2005
***
* **
• After learning: decrease of reported pain intensity for same stimulus decrease of evoked activity in ACC
Real-time fMRI and neuro-feedback
Beyond BOLD amplitude:multivoxel pattern analysis
15
Brain “reading” and pattern classification
Haynes et al., NN 2006 Kamitani et al., NN 2006Kay et al., Nature 2008
Accurate classification of new image: 92% correct out of 120, 82% out of 1000
(n= 1750)
Pattern classification analysis in FFA
?=
?=
- e.g. support vector machines, etc…
Faces VoicesBodies
Banse & Scherer, 1996 Atkinson et al., 2004 Belin et al., 2003
AngerFearDisgustSadnessJoy
vs Neutral
X
FFA pSTS EBA TVA
n=16
Emotions can be expressed in multiple modalities
all modulated by basic emotionsselective for stimulus category
16
vs ?
Emotion-specific patterns in “voice area”
Curr Biol 2009; 19(12):1028-33
Multi-voxel pattern analysis (MVPA)
and classification
Faces VoicesBodies
Banse & Scherer, 1996 Atkinson et al., 2004 Belin et al., 2003
AngerFearDisgustSadnessJoy
vs Neutral
X
Emotions can be expressed in multiple modalities
n=16
AmygdalaMPFC
all emotions > neutralin all modalities
Can we predict emotion in one modalitybased on pattern of response to another?
Supra-modal representations
Ventromedial prefrontal cx(vmPFC)
Correlation with same emotion in other modalityvs. correlation with other emotions
?
17
Supra-modal representations
Ventromedial prefrontal cx(vmPFC)
But no difference in the amygdala for correlations within vs. between emotions:
- insufficient resolution? - overlapping representations?
Correlation with same emotion in other modalityvs. correlation with other emotions
?
*
*
(*) average across F-V, F-B, and V-B pairs: p < .001
Supra-modal representations
right MPFC
left pSTS
But no difference in the amygdala for correlations within vs. between emotions:
- insufficient resolution? - overlapping representations?
Pairwise correlations and confusions
(across the 3 modalities)
r values
18
Conclusions
• It is possible to « decode » emotion category fromdistribution of cortical activity in modality-specificregions (e.g. faces, voices)
• Supra-modal representations of emotion exist in MPFC, partly reflecting valence but not arousal dimensions
• Their exact nature remains to be clarified (emotion specific or more general role in mental states attributions)
Beyond transient events:dynamic nature of emotions
and whole brain network approaches
Lasting impact of emotions on brain states
• Emotions are not only ¨acute¨ and transient responses to external stimuli
But…
• can also have long-lastingeffects on mental and bodily states
19
Functional network architecture identifiedthrough spontaneous fluctuations at rest
D. Zhang et al. (2010) Nature Reviews Neurology 6 , 15‐28.
- MPFC, PCC, Precuneus, IPL- rest > active task- self reflective processes, memory, interoceptive awareness and homeostasis
How is this “resting” activity perturbed by (transient) emotions?
+
restfilm
+
restfilm
Brain at (un)restafter emotional movies
+
rest
Post-FEAR<
Post-NEUTRAL
Post-JOY<
Post-NEUTRAL
T-value0
2
4
6
vMPFC
Precuneus
0
2
4
6
T-value
dACC
dMPFC
Precuneus
Neutral
Fearful
Joyful
Effects of prior emotions (induced by movies)
Neuroimage 2011, 54(3):2481-91
film
- on subsequent rest
film
film
Neutral
Fearful
Joyful
film
Effects of prior emotions (induced by movies)
++
++
++
FA FU NA NU-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
% S
igna
l cha
nge
NegativePositiveNeutral
FA FU NA NUAttended AttendedIgnored Ignored
Fearful faces Neutral faces
Movies
- on subsequent task
Faces:fearfulor neutral
left amygdala[-24, -6, -26]
R
Fearful > Neutral faces
p<.001
20
PCC-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5movies resting
Insula-2
-1.5
-1
-0.5
0
0.5
1
1.5movies resting
fearful
joyful
neutral
Precuneus-1.5
-1
-0.5
0
0.5
1
1.5restingmovies
Differential recovery of resting activity
3 successive time-bins(90 sec each)
ACC-2
-1.5
-1
-0.5
0
0.5
1
1.5movies resting
(and vmPFC)
Neuroimage 2011, 54(3):2481-91
Dynamic functional connectivity
time
0 5 10 min ROIs = 1 to n
RO
Is =
1 t
o n
BOLD fMRI
wavelet coefficient correlation(different frequency subbands)
individual ROIs (e.g. AAL atlas)
Neuroimage 2011, 56(2):616-26
Neuroimage 2011, 54(3):2481-91
Emotional effect on resting functional connectivity
• Post FEAR restingACC and PCC more connected to insula;
insula also more connected to IPL,thalamus, and pallidum.
Amygdala less coupled to VMPFC and precuneus.
• Post JOY resting
ACC and PCC more connected to IPL;
insula also more connected to IPL, thalamus, and pallidum
Decreased correlation in post-fear rest
Incr
ease
d co
rrel
atio
n in
pos
t-fe
ar r
est **
**** **** *
** *
*
** ** 0
0.5
1
1.5
2
2.5
3
3.5Ins_LIns_RACCPCC
IPL_LIPL_RPrec_LPrec_R
vMPFCAmy_LAmy_RThal_LHypoth
Pal_Rz-value
Decreased correlation in post-joy rest
Incr
ease
d co
rrel
atio
n in
pos
t-jo
y re
st
** ** *
** *
** * 0
0.5
1
1.5
2
2.5
3
3.5Ins_LIns_RACCPCC
IPL_LIPL_RPrec_LPrec_RvMPFCAmy_LAmy_RThal_LHypoth
Pal_Rz-value
• Higher subband (0.06-0.11 Hz) • Lower subband (0.03-0.06 Hz)
21
Prec
Amy
ACC vmPFC
25 30 35 40 45 50-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
STAI-T score
Nor
mal
ized
cor
rela
tion
diffe
renc
e
Precuneus - Amygdala
p< .00085
25 30 35 40 45 50-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
ACC - Amygdala
STAI-T score
Nor
mal
ized
cor
rela
tion
diff
eren
ce
p< .0051
Reduced connectivitypost FEAR > post JOY movies
individual differences in regulation processesoperating after emotional eventsare closely related to anxiety tendencies
NB: no correlation with absolute activation magnitude in single regions
Changes in functional connectivity are determined by individual anxiety levels
Shifts in amygdala reactivityare amplified by state anxiety
L R
Post positive movies Post negative movies
Conclusion
• Transient emotions modulate subsequent activity in brain networks associated with the “default mode”
- reduction in self-related thinking after emotion induction- associated with decreased activity in vMPFC (post fear > joy)
• Insula-centered network in post-fear and post-joy resting
- enhanced connectivity with ACC (post fear) and IPL (post joy), but also PCC, thalamus, and striatum- consistent with role in feeling states and emotion regulation
• Functional disturbances in same networks associated with anxiety
22
0.25-0.50 Hz 0.10-0.25 Hz 0.06-0.10 Hz 0.03-0.6 Hz
A
B
Functional connectivity patterns
• Network summary statistics
• Differences between states
• Pattern classification (e.g. SVM)
frontal
occipital
*
Different frequency sub-bands
Decoding affective statesfrom network connectivity patterns
Fear?
Neutral?
Joy?
+
rest
+
rest
Decoding affective state at rest
sample
sam
ple
transformed dissimilarity space
10 20 30 40 50 60
10
20
30
40
50
60
5
10
15
20
25
30
35
Rest post JOY
Rest post FEAR
connectivity matrices for 90 regions (computed at 0.03-0.11 Hz)
dissimilarity vector relative to neutral condition (direct embedding)
pattern classification analysis (decision tree)
4005 connections x 3 states
(joy, fear, neutral)
Dissimilaritybetween graphs
classificationmovie
movie
23
+
rest
+
rest
Decoding affective state at rest
sample
sam
ple
transformed dissimilarity space
10 20 30 40 50 60
10
20
30
40
50
60
5
10
15
20
25
30
35
Rest post JOY
Rest post FEAR
connectivity matrices for 90 regions (computed at 0.03-0.11 Hz)
dissimilarity vector relative to neutral condition (direct embedding)
pattern classification analysis (decision tree)
47/60 correctly classified(78.3%)
44/60 correctly classified(73.3%)
movie
movie
n= 15 subjects,4 epochs tested for each emotion
Differential importance of connections
Rest post FEAR vs JOYvmPFC-Amy_L
Insula_L-IPL_L Insula_L-Amy_R
vmPFC-Amy_R
vmPFC-IPL_R
WORK IN PROGRESS
Different configuration of functional networkscontaining partly similar regions
Obsessive-Compulsive Disorder (OCD)
• Chronic anxiety disorder
• Frequent: 2-3% prevalence
• Characterized by
- intrusive thoughts (obsessions) producing uneasiness, apprehension, fear, or worry; and- repetitive behaviors (compulsions) aimed at reducing anxiety.
Anomalies in spontaneous brain activity and connectivityeven “at rest”?
24
CaudateSMA
Amy
SPL
MidFC
IPL
IPL
Insula
MidFC
vMPFC
ACC
Caudate
Insula
*** *
* **
**
** ***
*** *
*
0
0.5
1
1.5
2
2.5
3
3.5Caudate_LCaudate_R
SMANuc_ACC_LNuc_ACC_R
ACCvMPFC_LdLPFC_LdLPFC_R
lOFC_LlOFC_RInsula_LInsula_R
Amy_LAmy_RSPL_LSPL_RIPL_LIPL_R
Mid_FC_LMid_FC_R
Prec_RSTG_LSTG_R
CO
NT
RO
LS
vs.
OC
D
OCD (n=16) vs CONTROLS (n=16)
Brain connectivity at (un)rest in OCD
• Enhanced connectivity of striatum and amygdala with fronto-limbic regions (VMPFC), repetitive intrusion of obsessive thoughts during resting
• Enhanced connectivity between premotor (SMA, midFC) and attention control areas (ACC, IPL) top-town regulation and inhibition of compulsive actions
Fast band (0.14-0.28 Hz)Unmedicated
Resting functional connectivity and OCD severity
6 8 10 12 14 16 18-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Obsession YBOCS score
Norm
alized
functional connectivity
Coupling between vmPFC and left caudate (free mind wandering condition)
r = 0.55p = 0.028
CaudateSMA
Amy
SPL
MidFC
IPL
IPL
InsulaMidFC
vMPFC
ACC
Caudate
Insula
25
Brain reading exerciceTypical fMRI activation to Music vs. Control Sound condition (30s epochs):
0
2
4
6
X=27 Y=-18
X=0
X=12 Y=9
Z=-3
Ventr. striatumRetrospl. cortexHPC
rACC
Visualcortex
Motor cortex
HPC
Pleasant affective evaluation
Memory and navigation
Visual imagery
Motor imagery
a mental “ballad” ? ...
• Music can induce emotions - but not basic categories!
• Psychological models suggest a specific domain of musical or “aesthetic” emotions…
• e.g. Zentner et al. 2008: 9 typical categories
Premotorcortex
STGSTG
CaudatePulvinar
N= 15 listeners,SPMs p ≤ 0.001
Z=-3
STG
Ventr.striatum Insula
VTA
Ventr.striatum
Hippoc
Sensory cortex
Ventr.striatum
HPC
VMPFC
Left HPC
Right HPC
HPC
PHG
VMPFC
PrecuneusDMPFC
Listening to 27 musical pieces (classic), 30-40 sec each,emotional ratings on 9 affective categories after each piece,
during fMRI
Factor analysis of subjective ratings
26
Z=-18
X=-3
Right hippocampus
Subgenual ACC
TENSPOW JOY WONTRATENDPEA NOS0
0.4
0.8
1.2
Mean xyz = 24.6, -13.4, -16.6Cluster size = 21 voxels
SAD
0
0.4
0.8
1.2
Mean xyz = -2.8, 31.6, -4.2Cluster size = 41 voxels
TENSPOW JOY WONTRATENDPEA NOS SAD
Y=15
Left caudate head
Right premotor cortex
-0.6
-0.2
0
0.2
0.6Mean xyz = 6.8, 16.7, 2Cluster size = 23 voxels
-0.4
0
0.4
0.8
1.2
Mean xyz = 61.7, 1.7, 31.5Cluster size = 14 voxels
TENSPOW JOY WON TRATENDPEA NOS SAD
TENSPOW JOY WON TRATENDPEA NOS SAD
Z=30
Differentiation between emotion categories
Main A+V- Main A-V+
– Affective space of musical emotions is consistent with the two basic dimensions of valence and arousal
– But more specific components seem related to memory (hippocampus), reflective processes (vmPFC), or motor circuits
– Finer distinction within each general emotion type may reflect a relative difference in the recruitment of similar components
Conclusions
Functional brain imaging can now be used to
• delineate circuits activated by various emotions (basic or more complex)
• determine factors that modulate such activations (stimulus features, task demands or context, personality, genes, psychopathology)
• guide new models and new therapeutic interventions
• test psychological theories or address philospohycontroversies (perhaps)
In summary
27
But research on emotions is still in its infancy,much remains to be learned…
Thank you for your attention !
Rationalist Model:
Emotions in moral judgments and decisions
Social Intuitionist Model:
Jonathan Haidt (2001).Psych Review, 108, 814-834.“The emotional dog and its rational tail: A social intuitionist approach to moral judgment„
Intuition = judgment including valence
(good-bad, like-dislike), made without effortful
steps of searching, pondering, or inferring
(unlike reasoning).
28
The « Trolley Problem »Two different situations in which you can sacrifice one human life to save five
Situation 1: ok
Situation 2: not ok
Science 2001, 293: 2105-2108.
• Both are social and "self-conscious" emotions
• Distinction remain controversial in psychology andphilosophy (Teroni & Deonna, 2008)
• Guilt is linked to transgressions of norms and might rely onrepresentations of others (i.e. victim of own misbehavior) –unlike shame (Baumeister et al., 1994).
• Shame might entail a stronger self-focus and be characterizedby subjective devaluation of the self – unlike guilt (e.g. Tangney 1999)
Moral emotions: Guilt and shame
* What are the common / distinct neural substrates? * Is there any difference in self- or other-referential processing?
Moral emotions: Guilt and shame
• Autobiographical memory paradigm in the fMRI scanner
• Based on private keywords (indirect questionnaire prior to scanning)
• Participants asked to relive episodes associated with:
- Guilt- Shame- Sadness- Neutral
(but emotion was not named)
lat OFC
General role in the retrieval of emotional memories
[Guilt+Shame+Sadness] > Neutral
0
2
4
6
8
10
dmPFCvmPFC
retrosplenial cx
visual cx
temporal pole anterior insula
29
Guilt > (Shame + Sadness)
Guilt-specific activation
Be
ta E
stim
ate
s (A
rbitr
ary
Uni
ts)
-0,2
-0,1
0,0
0,1
0,2
0,3
Guilt Shame Sadness Neutral
0
1
2
3
4
R lateral OFC L dmPFC
Trait Guilt Score
1,8 2,0 2,2 2,4 2,6 2,8 3,0 3,2 3,4 3,6 3,8 4,0
Be
tas
Est
ima
tes
for
Gu
ilt (
Arb
itrar
y U
nits
)
-0,6
-0,4
-0,2
0,0
0,2
0,4
0,6
0,8
r = 0.75p = 0.001
Guilt Inventory Questionnaire (Jones et al.,2000):= individual proneness to experience guilt
(e.g. “I worry a lot about things I have done in the past”)
R OFC
No shame-specific activation (shame > others)
Cereb Cor (2011)
Sociopathy and the historical case of Phineas Gage (1848)
Overlap with guilt
Overlap with both guilt and shame
Representations of self and other
Me
Me
Pessimistic
1 2 3 4 Me
Punctual
1 2 3 4 Me
Shy
1 2 3 4
Self > Other
…
Other
Other
Pessimistic
1 2 3 4 Other
Punctual
1 2 3 4 Other
Shy
1 2 3 4
…
Other > Self
Cereb Cor (2011)
30
Conclusion
• Both emotions recruit other-related representations (in STS, TPJ, retrosplenial cx): both are « social emotions »
• In addition, guilt also recruits self-related representations (in rostral ACC, ant insula): violations of norms as « internalized personal values »
• Guilt-specific activation in lateral OFC is unrelated to self or other: more general role in prediction of affective outcomes