58
IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D.

Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Rise of the machines(in MR image reconstruction)

Florian Knoll

NYU

Minnesota 2019 A.D.

Page 2: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Cooperations

Tom Pock & CoNIH R01 EB024532

Page 3: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

ML in radiology

ML

Page 4: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

ML for image reconstruction

ML

Page 5: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

ML for image reconstruction

ML

Page 6: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

ML for image reconstruction

ML

Page 7: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

ML for image reconstruction

ML

Page 8: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

MR data acquisition: Fourier (k-) space

Page 9: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Image reconstruction: Inverse problem

Page 10: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Image reconstruction: R=2

Page 11: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Non-Cartesian

Page 12: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Compressed sensing: Sparse representation

Lustig MRM (2007)

Page 13: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Compressed sensing: Sparse representation

Total Variation (TV)

Lustig MRM (2007)Rudin (1992)Block MRM (2007)

Page 14: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Numerical implementation

GD1 GDt GDT

uT

reconstruction

input

f

u0

Landweber Amer J Math 1951

Page 15: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Fully sampled Zero-filling R=4 • Separate artifacts from image content

• Sparsifying transform → Spatial filterkernels

• L1 norm → Potential functions

Hammernik ISMRM 2016, MRM 2018

CS → machine learning image reconstruction

Page 16: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Learning the numerical optimization

GD1 GDt GDT

uT

reconstruction

input

f

u0Learn T gradient descent (GD) steps

Page 17: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

GD1 GDt GDT uT

reconstructioninput

ut-1 ut

+

-

+

+

-

f

A “variational network”

Hammernik ISMRM 2016Hammernik MRM 2018

Page 18: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

xx

Learning for image reconstruction

parameters

reconstructioninput

reference

reconstruction error

similarity measure

Hammernik MRM 2018

Reconstruction model

Page 19: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Training setup• 3T knee exams, 15 channel knee coil, 5 sequences

• Cartesian subsampling: R=4, 24 reference lines

• ESPIRiT for coil sensitivity estimation

• 10 cases for training, 10 for testing

• 10 network stages

• 24 real and imaginary filter kernels, size 11x11

• Total number of parameters: 131050

• iPalm optimizer, 150 epochs, batch size 5

Hammernik MRM 2018Uecker MRM 2014Pock SIAM J Imag Sci 2016 https://github.com/VLOGroup/mri-variationalnetwork

Page 20: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

GD1 GDt GDT uT

reconstructioninput

f

Reconstructing new test data

Page 21: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Some reconstruction examples, R=4CG SENSE PI-CS: TGV

Learning: VN

Reference

corPD

M50

corPDFSF57

sagPD

F15

sagT2FSM34

axT2FSF45

M50

PI PI-CS Learning

F57

Hammernik, MRM 2018

Page 22: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

PI-CSPI Learning

Small fissure in tibial cartilage, R=4

Hammernik, ISMRM 2016

Page 23: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Abdominal Imaging: Clinical Study

23

Improved Image Sharpness

Compressed Sensing Deep Learning

Chen Radiology 2018 Slides courtesy of Joseph Cheng (Stanford)

Page 24: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Free breathing 3D whole heart coronary angiography

- Average acquisition time (m:s) was 18:55 for the fully sampled acquisition and 4:11 for an acceleration of 5x.- Average reconstruction time was ~5 minutes for CS and ~20 seconds for the VNN framework.

Slides courtesy of Claudia Prieto (Kings College)

Page 25: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Reconstruction of low dose CT data

Kobler IEEE ICASSP (2018)

Full

dose

¼ d

ose

ML

reco

n

Page 26: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Continuous radial DCE breast cancer MRIFrame 1 Frame 12

NU

FFT

VN 2

D

Frame 5

Page 27: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Continuous radial DCE breast cancer MRIFrame 1 Frame 12

NU

FFT

VN 2

DVN

2D

+t

Frame 5

Page 28: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Let’s look at some properties of our models

Page 29: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Reconstruction vs Image Processing

Initial Solution

Denoising Network

Reconstruction NetworkReference

SSIM=0.76

RMSE=0.22

SSIM=0.80

RMSE=0.19

SSIM=0.85

RMSE=0.14

Reference Zero-filling

0.76

Hammernik and Knoll, MICCAI Handbook 2019

Page 30: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Reconstruction vs Image Processing

Initial Solution

Denoising Network

Reconstruction NetworkReference

SSIM=0.76

RMSE=0.22

SSIM=0.80

RMSE=0.19

SSIM=0.85

RMSE=0.14

Reference Zero-filling VN postprocessing

0.76 0.80

Hammernik and Knoll, MICCAI Handbook 2019

Page 31: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Reconstruction vs Image Processing

Initial Solution

Denoising Network

Reconstruction NetworkReference

SSIM=0.76

RMSE=0.22

SSIM=0.80

RMSE=0.19

SSIM=0.85

RMSE=0.14

Reference Zero-filling VN postprocessing VN reconstruction

0.800.76 0.85

Hammernik and Knoll, MICCAI Handbook 2019

Page 32: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Systematic changes in forward modelTest case sampling Regular training Random training

Reg

ular

Ran

dom

Knoll MRM 2019

Page 33: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Systematic changes in forward modelTest case sampling Regular training Random training

Reg

ular

Ran

dom

Knoll MRM 2019

Page 34: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Systematic changes in forward modelTest case sampling Regular training Random training Joint training

Reg

ular

Ran

dom

Knoll MRM 2019

Page 35: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Training PDw FS Training PDw noise matched

Test

PD

wTe

st P

Dw

FSRobustness and generalization: Contrast/SNR

Knoll MRM 2019

Training PDw Joint training

0.94 0.91 0.940.91

0.81 0.89 0.890.88

Page 36: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Robustness and generalization: SequencesIn

divi

dual

trai

ning

Join

t tra

inin

g

Coronal PDwAxial T2w Coronal PDw FS Sag T2w FSSag PDw

0.93 0.97 0.970.91

0.93 0.97 0.970.91

0.93

0.93

Clinical study: Manuscript in preparation

Page 37: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Robustness and generalization: Anatomy

0.90 0.92 0.97 0.90 0.92

0.89 0.91 0.97 0.89 0.91

Knoll ISMRM (2018)

Indi

vidu

al tr

aini

ngJo

int t

rain

ing

Page 38: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019Antun et al. arXiv (2019)

Page 39: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019Antun et al. arXiv (2019)

Page 40: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019Antun et al. arXiv (2019)

Page 41: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019Antun et al. arXiv (2019)

Page 42: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

200 training slices≈130K model parameters

R=4 reconstructionGround truth

Hammernik MRM 2018

Data size and model complexity

Page 43: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

≈50000 training slices≈30M model parameters

R=4 reconstructionGround truth

Data size and model complexity

fastmri.med.nyu.eduKnoll Radiology AI (2019, in press)

Page 44: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Influence of the training loss functions

Page 45: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Iterative (CG) sense reconstruction

RMSE to reference over iterations

RMSE optimal: 11 iterations Radiologist selection: 18 iterationsFully sampled referenceRMSE to fully sampled reference

A simple iterative SENSE experiment

Pruessman MRM (2001)

600 spokes55 spokes subsampling16 receive channels

Page 46: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Iterative (CG) sense reconstruction

RMSE to reference over iterations

RMSE optimal: 11 iterations Radiologist selection: 18 iterationsFully sampled referenceRMSE to fully sampled reference

A simple iterative SENSE experiment

Pruessman MRM (2001)

600 spokes55 spokes subsampling16 receive channels

Page 47: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019Narnhofer SPIE 2019

nrmse [0.1834] ssim [0.7009]

Zero Filling

nrmse [0.0940] ssim [0.7896]

CG SENSE

nrmse [0.0583] ssim [0.8874]

VN

nrmse [0.0658] ssim [0.8567]

GAN Reference

R=6 reconstruction with GAN priorVN GAN prior Reference

Page 48: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Diagnostic classification

Predict presence of malignant and/or benign lesions in each breast

• 5000 training• 1500 validation• 1500 validation

Gong NeurIPS Medical Imaging (2019)

Page 49: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

Page 50: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

Page 51: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

Page 52: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Diagnostic classification

Gong NeurIPS Medical Imaging (2019)

Shen MLMI (2019)

Page 53: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

End to end reconstruction and classification

undersampled k-space !

fully sampled k-space

Reconstruction network "

Fully sampled reference #$%&,(

Reconstruction output #()

**

Image based error metric

Update parameters +,

PredictionLeft Malignant: [0,1]

Left Benign: [0,1]Right Malignant: [0,1]

Right Benign: [0,1]

Classification network -

Update parameters +.

+

LabelLeft Malignant: [0,1]

Left Benign: [0,1]Right Malignant: [0,1]

Right Benign: [0,1]

Page 54: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

End to end reconstruction and classificationSeparate End-to-end

Work in progress :-)

Page 55: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

Discussion

• Robust enough for clinical routine?

• Integration in clinical workflow?

• Validation: When and how do things go wrong?

• Theory!

Page 56: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

ChallengeSeptember 19th: Challenge submission deadline

34 submissions

Evaluation completed, winners still secret :-)– Phase 1: Quantitative metrics– Phase 2: Radiologists scoring

Dezember 13th to 14th: Oral presentations at NeurIPS medical imaging

https://nips.cc/https://sites.google.com/view/med-neurips-2019

Page 57: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

AcknowledgementsNYU

– Mary Bruno– Krzysztof Geras– Shizhan Gong– Zhengnan Huang– Patricia Johnson– Linda Moy– Gene Kim– Matt Muckley– Michael Recht– Dan Sodickson– Ruben Stern

TU Graz– Kerstin Hammernik (now at Imperial)– Erich Kobler– Dominik Narnhofer– Thomas Pock

Facebook AI research– Tullie Murrell– Anuroop Sriram– Nafissa Yakubova– Jure Zbontar– Larry Zitnick

Grant Support– NIH R01 EB024532– NIH P41 EB017183– NIH R21 EB027241– Amazon

Page 58: Rise of the machines · 2019. 10. 23. · IMA CI 17/10/2019 Rise of the machines (in MR image reconstruction) Florian Knoll NYU Minnesota 2019 A.D

IMA CI 17/10/2019

http://cai2r.net/people/florian-knoll

Postdoc positions available: https://cai2r.net/jobs/