IMA CI 17/10/2019

Rise of the machines(in MR image reconstruction)

Florian Knoll

NYU

Minnesota 2019 A.D.

IMA CI 17/10/2019

Cooperations

Tom Pock & CoNIH R01 EB024532

IMA CI 17/10/2019

ML in radiology

ML

IMA CI 17/10/2019

ML for image reconstruction

ML

IMA CI 17/10/2019

ML for image reconstruction

ML

IMA CI 17/10/2019

ML for image reconstruction

ML

IMA CI 17/10/2019

ML for image reconstruction

ML

IMA CI 17/10/2019

MR data acquisition: Fourier (k-) space

IMA CI 17/10/2019

Image reconstruction: Inverse problem

→

IMA CI 17/10/2019

Image reconstruction: R=2

→

IMA CI 17/10/2019

Non-Cartesian

→

IMA CI 17/10/2019

Compressed sensing: Sparse representation

Lustig MRM (2007)

IMA CI 17/10/2019

Compressed sensing: Sparse representation

Total Variation (TV)

Lustig MRM (2007)Rudin (1992)Block MRM (2007)

IMA CI 17/10/2019

Numerical implementation

GD1 GDt GDT

uT

reconstruction

input

f

u0

Landweber Amer J Math 1951

IMA CI 17/10/2019

Fully sampled Zero-filling R=4 • Separate artifacts from image content

• Sparsifying transform → Spatial filterkernels

• L1 norm → Potential functions

Hammernik ISMRM 2016, MRM 2018

CS → machine learning image reconstruction

IMA CI 17/10/2019

Learning the numerical optimization

GD1 GDt GDT

uT

reconstruction

input

f

u0Learn T gradient descent (GD) steps

IMA CI 17/10/2019

GD1 GDt GDT uT

reconstructioninput

ut-1 ut

+

-

+

+

-

f

A “variational network”

Hammernik ISMRM 2016Hammernik MRM 2018

IMA CI 17/10/2019

xx

Learning for image reconstruction

parameters

reconstructioninput

reference

reconstruction error

similarity measure

Hammernik MRM 2018

Reconstruction model

IMA CI 17/10/2019

Training setup• 3T knee exams, 15 channel knee coil, 5 sequences

• Cartesian subsampling: R=4, 24 reference lines

• ESPIRiT for coil sensitivity estimation

• 10 cases for training, 10 for testing

• 10 network stages

• 24 real and imaginary filter kernels, size 11x11

• Total number of parameters: 131050

• iPalm optimizer, 150 epochs, batch size 5

Hammernik MRM 2018Uecker MRM 2014Pock SIAM J Imag Sci 2016 https://github.com/VLOGroup/mri-variationalnetwork

IMA CI 17/10/2019

GD1 GDt GDT uT

reconstructioninput

f

Reconstructing new test data

IMA CI 17/10/2019

Some reconstruction examples, R=4CG SENSE PI-CS: TGV

Learning: VN

Reference

corPD

M50

corPDFSF57

sagPD

F15

sagT2FSM34

axT2FSF45

M50

PI PI-CS Learning

F57

Hammernik, MRM 2018

IMA CI 17/10/2019

PI-CSPI Learning

Small fissure in tibial cartilage, R=4

Hammernik, ISMRM 2016

IMA CI 17/10/2019

Abdominal Imaging: Clinical Study

23

Improved Image Sharpness

Compressed Sensing Deep Learning

Chen Radiology 2018 Slides courtesy of Joseph Cheng (Stanford)

IMA CI 17/10/2019

Free breathing 3D whole heart coronary angiography

- Average acquisition time (m:s) was 18:55 for the fully sampled acquisition and 4:11 for an acceleration of 5x.- Average reconstruction time was ~5 minutes for CS and ~20 seconds for the VNN framework.

Slides courtesy of Claudia Prieto (Kings College)

IMA CI 17/10/2019

Reconstruction of low dose CT data

Kobler IEEE ICASSP (2018)

Full

dose

¼ d

ose

ML

reco

n

IMA CI 17/10/2019

Continuous radial DCE breast cancer MRIFrame 1 Frame 12

NU

FFT

VN 2

D

Frame 5

IMA CI 17/10/2019

Continuous radial DCE breast cancer MRIFrame 1 Frame 12

NU

FFT

VN 2

DVN

2D

+t

Frame 5

IMA CI 17/10/2019

Let’s look at some properties of our models

IMA CI 17/10/2019

Reconstruction vs Image Processing

Initial Solution

Denoising Network

Reconstruction NetworkReference

SSIM=0.76

RMSE=0.22

SSIM=0.80

RMSE=0.19

SSIM=0.85

RMSE=0.14

Reference Zero-filling

0.76

Hammernik and Knoll, MICCAI Handbook 2019

IMA CI 17/10/2019

Reconstruction vs Image Processing

Initial Solution

Denoising Network

Reconstruction NetworkReference

SSIM=0.76

RMSE=0.22

SSIM=0.80

RMSE=0.19

SSIM=0.85

RMSE=0.14

Reference Zero-filling VN postprocessing

0.76 0.80

Hammernik and Knoll, MICCAI Handbook 2019

IMA CI 17/10/2019

Reconstruction vs Image Processing

Initial Solution

Denoising Network

Reconstruction NetworkReference

SSIM=0.76

RMSE=0.22

SSIM=0.80

RMSE=0.19

SSIM=0.85

RMSE=0.14

Reference Zero-filling VN postprocessing VN reconstruction

0.800.76 0.85

Hammernik and Knoll, MICCAI Handbook 2019

IMA CI 17/10/2019

Systematic changes in forward modelTest case sampling Regular training Random training

Reg

ular

Ran

dom

Knoll MRM 2019

IMA CI 17/10/2019

Systematic changes in forward modelTest case sampling Regular training Random training

Reg

ular

Ran

dom

Knoll MRM 2019

IMA CI 17/10/2019

Systematic changes in forward modelTest case sampling Regular training Random training Joint training

Reg

ular

Ran

dom

Knoll MRM 2019

IMA CI 17/10/2019

Training PDw FS Training PDw noise matched

Test

PD

wTe

st P

Dw

FSRobustness and generalization: Contrast/SNR

Knoll MRM 2019

Training PDw Joint training

0.94 0.91 0.940.91

0.81 0.89 0.890.88

IMA CI 17/10/2019

Robustness and generalization: SequencesIn

divi

dual

trai

ning

Join

t tra

inin

g

Coronal PDwAxial T2w Coronal PDw FS Sag T2w FSSag PDw

0.93 0.97 0.970.91

0.93 0.97 0.970.91

0.93

0.93

Clinical study: Manuscript in preparation

IMA CI 17/10/2019

Robustness and generalization: Anatomy

0.90 0.92 0.97 0.90 0.92

0.89 0.91 0.97 0.89 0.91

Knoll ISMRM (2018)

Indi

vidu

al tr

aini

ngJo

int t

rain

ing

IMA CI 17/10/2019Antun et al. arXiv (2019)

IMA CI 17/10/2019Antun et al. arXiv (2019)

IMA CI 17/10/2019Antun et al. arXiv (2019)

IMA CI 17/10/2019Antun et al. arXiv (2019)

IMA CI 17/10/2019

200 training slices≈130K model parameters

R=4 reconstructionGround truth

Hammernik MRM 2018

Data size and model complexity

IMA CI 17/10/2019

≈50000 training slices≈30M model parameters

R=4 reconstructionGround truth

Data size and model complexity

fastmri.med.nyu.eduKnoll Radiology AI (2019, in press)

IMA CI 17/10/2019

Influence of the training loss functions

IMA CI 17/10/2019

Iterative (CG) sense reconstruction

RMSE to reference over iterations

RMSE optimal: 11 iterations Radiologist selection: 18 iterationsFully sampled referenceRMSE to fully sampled reference

A simple iterative SENSE experiment

Pruessman MRM (2001)

600 spokes55 spokes subsampling16 receive channels

IMA CI 17/10/2019

Iterative (CG) sense reconstruction

RMSE to reference over iterations

RMSE optimal: 11 iterations Radiologist selection: 18 iterationsFully sampled referenceRMSE to fully sampled reference

A simple iterative SENSE experiment

Pruessman MRM (2001)

600 spokes55 spokes subsampling16 receive channels

IMA CI 17/10/2019Narnhofer SPIE 2019

nrmse [0.1834] ssim [0.7009]

Zero Filling

nrmse [0.0940] ssim [0.7896]

CG SENSE

nrmse [0.0583] ssim [0.8874]

VN

nrmse [0.0658] ssim [0.8567]

GAN Reference

R=6 reconstruction with GAN priorVN GAN prior Reference

IMA CI 17/10/2019

Diagnostic classification

Predict presence of malignant and/or benign lesions in each breast

• 5000 training• 1500 validation• 1500 validation

Gong NeurIPS Medical Imaging (2019)

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

Shen MLMI (2019)

IMA CI 17/10/2019

End to end reconstruction and classification

undersampled k-space !

fully sampled k-space

Reconstruction network "

Fully sampled reference #$%&,(

Reconstruction output #()

**

Image based error metric

Update parameters +,

PredictionLeft Malignant: [0,1]

Left Benign: [0,1]Right Malignant: [0,1]

Right Benign: [0,1]

Classification network -

Update parameters +.

+

LabelLeft Malignant: [0,1]

Left Benign: [0,1]Right Malignant: [0,1]

Right Benign: [0,1]

IMA CI 17/10/2019

End to end reconstruction and classificationSeparate End-to-end

Work in progress :-)

IMA CI 17/10/2019

Discussion

• Robust enough for clinical routine?

• Integration in clinical workflow?

• Validation: When and how do things go wrong?

• Theory!

IMA CI 17/10/2019

ChallengeSeptember 19th: Challenge submission deadline

34 submissions

Evaluation completed, winners still secret :-)– Phase 1: Quantitative metrics– Phase 2: Radiologists scoring

Dezember 13th to 14th: Oral presentations at NeurIPS medical imaging

https://nips.cc/https://sites.google.com/view/med-neurips-2019

IMA CI 17/10/2019

AcknowledgementsNYU

– Mary Bruno– Krzysztof Geras– Shizhan Gong– Zhengnan Huang– Patricia Johnson– Linda Moy– Gene Kim– Matt Muckley– Michael Recht– Dan Sodickson– Ruben Stern

TU Graz– Kerstin Hammernik (now at Imperial)– Erich Kobler– Dominik Narnhofer– Thomas Pock

Facebook AI research– Tullie Murrell– Anuroop Sriram– Nafissa Yakubova– Jure Zbontar– Larry Zitnick

Grant Support– NIH R01 EB024532– NIH P41 EB017183– NIH R21 EB027241– Amazon

IMA CI 17/10/2019

http://cai2r.net/people/florian-knoll

Postdoc positions available: https://cai2r.net/jobs/