Generative Models of M/EEG: Group inversion and MEG+EEG+fMRI multimodal integration Rik Henson

Generative Models of M/EEG:

Group inversion and MEG+EEG+fMRI multimodal integration

Rik Henson

(with much input from Karl Friston)

Overview

1. A Generative Model of M/EEG

2. Group inversion (optimising priors across subjects)

3. Multimodal integration:

3.1 Symmetric integration (fusion) of MEG + EEG

3.2 Asymmetric integration of MEG + fMRI

3.3 Full fusion of MEG/EEG + fMRI?

1. A PEB Framework for MEG/EEG(Generative Model)

Phillips et al (2005), Neuroimage

Y = LJ +EY = Data n sensorsJ = Sources p>>n sourcesL = Leadfields n sensors x p sourcesE = Error n sensors

(Linear) Forward Model for MEG/EEG (for one timepoint):

| ( | ) ( )p( ) p pJ Y Y J J

(Gaussian) Likelihood:

( )( | ) ( , )ep NY J LJ C

( ) (0, )jp N ( )J C

C(e) = n x n Sensor (error) covariance

Prior:

C(j) = p x p Source (prior) covariance

Posterior:

( )eiQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

i ii

C Q C = Sensor/Source covarianceQ = Covariance componentsλ = Hyper-parameters

( )jiQ2. Source components,

(priors/regularisation):

“IID” (white noise):

# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

“IID” (min norm):

# sources

# s

ou

rce

s

Multiple Sparse Priors (MSP):

( ) ( , )p NX m C

# sources

# s

ou

rce

sFriston et al (2008) Neuroimage

1. A PEB Framework for MEG/EEG(Generative Model)

ΕJ

YL

( )jC( )eC

( , )N 0 C ( , )N 0 C

...

( )ji

( )1eQ ( )

2eQ ...

( )ei

1. A PEB Framework for MEG/EEG(Generative Model)

Friston et al (2008) Neuroimage

Fixed

Variable

Data

( )1jQ ( )

2jQ

1. A PEB Framework for MEG/EEG(Inversion)

Friston et al (2002) Neuroimage

ˆ max ( | ) maxp F

λ Y λ

ˆˆln ( | ) ln ( , | ) ( , )p m p J m dJ F Y Y J λ

ˆˆ max ( | , ) maxj jp F J J Y λ

ln ( | , ) ( | ) lnq

F p p q Y J λ J λ

1. Obtain Restricted Maximum Likelihood (ReML) estimates of the hyperparameters (λ) by maximising the variational “free energy” (F):

2. Obtain Maximum A Posteriori (MAP) estimates of parameters (sources, J):( ) ( ) ( ) -1ˆ ˆ ˆ( )j T j T eC L LC L +C Y cf. Tikhonov

…and an estimate of their posterior covariance (inverse precision):

( ) ( ) ( )ˆ ˆ ˆ ˆˆ -j j j T -1Σ C C L C LC

3. Maximal F approximates Bayesian (log) “model evidence” for a model, m:

(relevant to MEG+EEG integration)

(relevant to MEG+fMRI integration)

( )ˆ ˆ ˆj e T ( )C LC L C

ˆ{ , , }m L Q λ

Summary:

1. A PEB Framework for MEG/EEG

• Automatically “regularises” in principled fashion…

• …allows for multiple constraints (priors)…

• …to the extent that multiple (100’s) of sparse priors possible…

• …(or multiple error components or multiple fMRI priors)…

• …furnishes estimates of source precisions and model evidence

2. Group Inversion

( )eiQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

i ii

C Q C = Sensor/Source covarianceQ = Covariance componentsλ = Hyper-parameters

( )jiQ2. Source components,

(priors/regularisation):

“IID” (white noise):

# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

“IID” (min norm):

# sources

# s

ou

rce

s

Multiple Sparse Priors (MSP):

( ) ( , )p NX m C

# sources

# s

ou

rce

sFriston et al (2008) Neuroimage

( )eiQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

i ii

C Q C = Sensor/Source covarianceQ = Covariance componentsλ = Hyper-parameters

( )jiQ

“IID” (white noise):

# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

( ) ( , )p NX m C

2. Optimise Multiple Sparse Priors by pooling across participants

2. Group Inversion

Litvak & Friston (2008) Neuroimage

# sources

# s

ou

rce

s

ΕJ

YL

( )jC( )eC

( , )N 0 C ( , )N 0 C

...

( )ji

( )1eQ ( )

2eQ ...

( )ei

2. Group Inversion (single subject)(Generative Model)

Litvak & Friston (2008) Neuroimage

( )1jQ ( )

2jQ

ΕJ

1Y

iL

( )jC ( )eC

( , )N 0 C ( , )N 0 C

...

( )jk

( )21eQ

( )eik

2. Group Inversion (multiple subjects)(Generative Model)

Litvak & Friston (2008) Neuroimage

( )1jQ ( )

2jQ

2Y NY

( )12eQ ( )

1eNQ( )

11eQ ...

)()()( jk

jk

j QC

Ti

eki

eik

ei AQAC )()()(

)(0

)(0

eTj CLCLC

)(

)(2

)(1

)(

00

0

0

00

eN

e

e

e

C

C

C

C

NNNE

E

E

J

L

L

L

Y

Y

Y

2

1

2

1

2

1

~

~

~

iii YAY ~

…projecting data and leadfields to a reference subject (0):

Common source-level priors:

Subject-specific sensor-level priors:

10 )( T

iiTii LLLLA

2. Group Inversion(Generative Model)

Litvak & Friston (2008) Neuroimage

2. Group Inversion(Generative Model)

Litvak & Friston (2008) Neuroimage

MMN MSP MSP (Group)

fMRI MEG ? (future)Data:

Causes (hidden):

Generative (Forward)Models:

BalloonModel

HeadModel

?

EEG

HeadModel

“Neural”Activity

3. Types of Multimodal Integration

(inversion)

AsymmetricIntegration

fMRI MEG ? (future)Data:

Causes (hidden):

Generative (Forward)Models:

BalloonModel

HeadModel

?

EEG

HeadModel

“Neural”Activity

SymmetricIntegration(Fusion)

3. Types of Multimodal Integration

Daunizeau et al (2007), Neuroimage

( )eiQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

i ii

C Q C = Sensor/Source covarianceQ = Covariance componentsλ = Hyper-parameters

( )jiQ2. Source components,

(priors/regularisation):

“IID” (white noise):

# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

“IID” (min norm):

# sources

# s

ou

rce

s

Multiple Sparse Priors (MSP):

( ) ( , )p NX m C

# sources

# s

ou

rce

sFriston et al (2008) Neuroimage

3.1 Fusion of MEG+EEG(Theory)

( )eijQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

Ci(e) = Sensor error covariance for ith modality

Qij = jth component for ith modalityλij = Hyper-parameters

( )jiQ2. Source components,

(priors/regularisation):

“IID” (min norm):

# sources

# s

ou

rce

s

Multiple Sparse Priors (MSP):

# sources

# s

ou

rce

s

( ) ( ) ( )e e ei ji ij

j

C Q

# sensors

# s

en

sors

# sensors

# s

en

sors

E.g, white noise for 2 modalities:( )11eQ ( )

21eQ

Henson et al (2009) Neuroimage

3.1 Fusion of MEG+EEG(Theory)

ΕJ

MEGY

MEGL

( )jC( )eC

( , )N 0 C ( , )N 0 C

( )1jQ ( )

2jQ

( )ji

( )1eQ ( )

2eQ

( )ei

3.1 Fusion of MEG+EEG(Generative Model)

Henson et al (2009) Neuroimage

ΕJ

MEGY

MEGL

( )jC( )1eC

( , )N 0 C ( , )N 0 C

( )1jQ ( )

2jQ

( )ji

( )11eQ ( )

12eQ

( )eij

3.1 Fusion of MEG+EEG(Generative Model)

EEGY

EEGL

( )2eC

( )21eQ ( )

22eQ

Henson et al (2009) Neuroimage

3.1 Fusion of MEG+EEG(Theory)

Henson et al (2009) Neuroimage

(1)11 1(1)22 2

(1)dd d

EY L

EY LJ

EY L

• Stack data and leadfields for d modalities:

1 ( )i

ii T

i im

YY

tr YY

• Where data / leadfields scaled to have same average / predicted variance:

)(

)(2

)(1

)(

00

0

0

00

ed

e

e

e

C

C

C

C

mi = Number of spatial modes (e.g, channels)1 ( )

i

ii T

i im

LL

tr L L

(note: common sources and source priors, but separate error components)

ERs from 12 subjects for 3 simultaneously-acquired Neuromag sensor-types:

RM

S f

T/m V

FacesScrambled

fT

3.1 Fusion of MEG+EEG(Application)

Magnetometers (MEG, 102)

(Planar) Gradiometers (MEG, 204)

Electrodes (EEG, 70)

Henson et al (2009) Neuroimage

150-190ms

Faces - Scrambled

ms ms ms

MEG mags MEG grads

EEG

FUSED

+31 -51 -15 +19 -48 -6

+43 -67 -11 +44 -64 -4

3.1 Fusion of MEG+EEG

Henson et al (2009) Neuroimage

ˆ1/ 59ii ˆ1/ 76ii

IID noise for each modality; common MSP for sources

ˆ1/ 95ii ˆ1/ 127ii

(fixed number of spatial+temporal modes)

Scrambled

150-190msFaces – Scrambled,

Faces

3.1 Fusion of MEG+EEG(Conclusions)

Henson et al (2009) Neuroimage

• Fusing magnetometers, gradiometers and EEG increased the conditional precision of the source estimates relative to inverting any one modality alone (when equating number of spatial+temporal modes)

• The maximal sources recovered from fusion were a plausible combination of the ventral temporal sources recovered by MEG and the lateral temporal sources recovered by EEG

• (Simulations show the relative scaling of mags and grads agrees with empty-room data)

3.2 Integration of M/EEG+fMRI

( )eiQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

i ii

C Q C = Sensor/Source covarianceQ = Covariance componentsλ = Hyper-parameters

( )jiQ2. Source components,

(priors/regularisation):

“IID” (white noise):

# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

“IID” (min norm):

# sources

# s

ou

rce

s

Multiple Sparse Priors (MSP):

( ) ( , )p NX m C

# sources

# s

ou

rce

sFriston et al (2008) Neuroimage

Henson et al (in press) Human Brain Mapping

( )eiQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

i ii

C Q C = Sensor/Source covarianceQ = Covariance componentsλ = Hyper-parameters

( )jiQ

“IID” (white noise):

# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

“IID” (min norm):

# sources

# s

ou

rce

s

fMRI Priors:

( ) ( , )p NX m C

# sources

# s

ou

rce

s

2. Each suprathreshold fMRI cluster becomes a separate prior

3.2 Integration of M/EEG+fMRI

3.2 Integration of M/EEG+fMRI(Generative Model)

ΕJ

YL

( )jC( )eC

( , )N 0 C ( , )N 0 C

...

( )ji

( )1eQ ( )

2eQ ...

( )ei

( )1jQ ( )

2jQ

3.2 Integration of M/EEG+fMRI(Generative Model)

ΕJ

L

( )jC( )eC

( , )N 0 C ( , )N 0 C

MEGY

fMRIY

( )1jQ ( )

2jQ

( )ji

( )ei

( )1eQ ( )

2eQ( )

3jQ ( )

4jQ

Henson et al (in press) Human Brain Mapping

T1-weighted MRI

Anatomical data

{T,F,Z}-SPM

Gray matter segmentation

Cortical surfaceextraction

3D geodesicVoronoï diagram

Functional data

…

1. Thresholding and connected component labelling

…

2. Projection onto the cortical surface using the Voronoï diagram

…

3. Prior covariance components ( )jiQ

3.2 Integration of M/EEG+fMRI (Priors)

3.2 Integration of M/EEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

SPM{F} for faces versus scrambled faces,

15 voxels, p<.05 FWE

5 clusters from SPM of fMRI data from separate group of (18) subjects in MNI space

1 2

3 4 5

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

(binarised, variance priors)

Magnetometers (MEG)

* *

* *

None Global Local (Valid) Local (Invalid) Valid+Invalid

Electrodes (EEG)

Neg

ativ

e F

ree

Ene

rgy

(a.u

.)(m

odel

evi

denc

e)

*

*

**

*

*

*

Gradiometers (MEG)

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

(binarised, variance priors)

Magnetometers (MEG)

* *

* *

Gradiometers (MEG)

None Global Local (Valid) Local (Invalid) Valid+Invalid

Electrodes (EEG)

Neg

ativ

e F

ree

Ene

rgy

(a.u

.)(m

odel

evi

denc

e)

*

*

**

*

*

*

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

(binarised, variance priors)

Magnetometers (MEG)

* *

* *

Gradiometers (MEG)

None Global Local (Valid) Local (Invalid) Valid+Invalid

Electrodes (EEG)

Neg

ativ

e F

ree

Ene

rgy

(a.u

.)(m

odel

evi

denc

e)

*

*

**

*

*

*

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

(binarised, variance priors)

Magnetometers (MEG)

* *

* *

Gradiometers (MEG)

None Global Local (Valid) Local (Invalid) Valid+Invalid

Electrodes (EEG)

Neg

ativ

e F

ree

Ene

rgy

(a.u

.)(m

odel

evi

denc

e)

*

*

**

*

*

*

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

(binarised, variance priors)

Magnetometers (MEG)

* *

* *

Gradiometers (MEG)

None Global Local (Valid) Local (Invalid) Valid+Invalid

Electrodes (EEG)

Neg

ativ

e F

ree

Ene

rgy

(a.u

.)(m

odel

evi

denc

e)

*

*

**

*

*

*

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

None Global Local (Valid) Local (Invalid)

Magnetometers (MEG)

Gradiometers (MEG)

Electrodes (EEG)

IID sources and IID noise (L2 MNM)

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

None Global Local (Valid) Local (Invalid)

Magnetometers (MEG)

Gradiometers (MEG)

Electrodes (EEG)

IID sources and IID noise (L2 MNM)

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

fMRI priors counteract superficial bias of L2-norm

None Global Local (Valid) Local (Invalid)

Magnetometers (MEG)

Gradiometers (MEG)

Electrodes (EEG)

IID sources and IID noise (L2 MNM)

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

fMRI priors counteract superficial bias of L2-norm

None Global Local (Valid) Local (Invalid)

Magnetometers (MEG)

Gradiometers (MEG)

Electrodes (EEG)

IID sources and IID noise (L2 MNM)

3.2 Fusion of MEG+fMRI (Application)

Henson et al (in press) Human Brain Mapping

NB: Priors affect variance, not precise timecourse…

R

L

Gradiometers (MEG)

(5 Local Valid Priors)

Diff

eren

tial R

espo

nse

(Fac

es v

s S

cram

bled

)

Diff

eren

tial R

espo

nse

(Fac

es v

s S

cram

bled

)

Right Posterior Fusiform (rPF) Right Medial Fusiform (rMF) Right Lateral Fusiform (rLF)

Left occipital pole (lOP)

-27 -93 0

+26 -76 -11 +41 -43 -24 +32 -45 -12

-43 -47 -21

Left Lateral Fusiform (lLF)

Diff

eren

tial R

espo

nse

(Fac

es v

s S

cram

bled

)

• Adding a single, global fMRI prior increases model evidence

• Adding multiple valid priors increases model evidence furtherHelpful if some fMRI regions produce no MEG/EEG signal (or arise from neural activity at different times)

• Adding invalid priors rarely increases model evidence, particularly in conjunction with valid priors

• Can counteract superficial bias of, e.g, minimum-norm

• Affects variance but not not precise timecourse

• (Adding fMRI priors to MSP has less effect)

3.2 Fusion of MEG+fMRI (Conclusions)

Henson et al (in press) Human Brain Mapping

3.3 Fusion of fMRI and MEG/EEG?

fMRI MEG ? (future)Data:

Causes (hidden):

BalloonModel

HeadModel

?

EEG

HeadModel

“Neural”Activity

Fusion of fMRI + MEG/EEG?

Henson (2010) Biomag

J

MEGY

( )sMEGL

( )jC

( )1jQ ( )

2jQ

MEGΕ

( )eC

( )1eQ ( )

2eQ

3.3 Fusion of fMRI and MEG/EEG?

Henson (2010) Biomag

J

MEGY

( )sMEGL

( )jC

( )1jQ ( )

2jQ

( )tfMRIH

MEGΕ

( )eC

( )1eQ ( )

2eQ

fMRIΕ

fMRIY

( )1tA ( )

2sQ

time (t)?space (s)

3.3 Fusion of fMRI and MEG/EEG?

Henson (2010) Biomag

Overall Conclusions

1. The PEB (in SPM8) framework is advantageous

2. Group optimisation of MSPs can be advantageous

3. Full fusion of MEG and EEG is advantageous

4. Using fMRI as (spatial) priors on MEG is advantageous

5. Unclear that fusion of fMRI and M/EEG is advantageous

The End

3. Fusion of MEG+EEG

Henson et al (2009) Neuroimage

Participant

Grads

Mags

log

(λx10

6 )

Participant

EE

G

Grads

Mags

log

(λx10

6 )

3. Fusion of MEG+EEGH

yper

pa

ram

ete

rs

Henson et al (2009) Neuroimage

4. Fusion of MEG+fMRI

Henson et al (in press) Human Brain Mapping

fMR

I hyp

erp

ara

me

ters

ln(λ

)+32

ln(λ

)+32

Participant

Participant

Magnetometers (MEG) Gradiometers (MEG) Electrodes (EEG)

Local Valid

Local Invalid