Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Metric learning for diffeomorphic imageregistration.
François-Xavier Vialard
Université Paris-Est Marne-la-Vallée
joint work with M. Niethammer and R. Kwitt.
IHP, March 2019.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Outline
1 Introduction to diffeomorphisms group and Riemannian tools
2 Choice of the metric
3 Spatially dependent metrics
4 Metric learning
5 SVF metric learning
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Example of problems of interest
Given two shapes, find a diffeomorphism of R3 that maps oneshape onto the other
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Example of problems of interest
Given two shapes, find a diffeomorphism of R3 that maps oneshape onto the other
Different data types and different way of representing them.
Figure – Two slices of 3D brain images of the same subject at differentages
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Example of problems of interestGiven two shapes, find a diffeomorphism of R3 that maps oneshape onto the other
Deformation by a diffeomorphism
Figure – Diffeomorphic deformation of the image
SimulationBrain.mp4Media File (video/mp4)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Variety of shapes
Figure – Different anatomical structures extracted from MRI data
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Variety of shapes
Figure – Different anatomical structures extracted from MRI data
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
A Riemannian approach to diffeomorphicregistration
Several diffeomorphic registration methods are available:
• Free-form deformations B-spline-based diffeomorphisms by D.Rueckert
• Log-demons (X.Pennec et al.)• Large Deformations by Diffeomorphisms (M. Miller,A.
Trouvé, L. Younes)
• ANTSOnly the two last ones provide a Riemannian framework.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
A Riemannian approach to diffeomorphicregistration
• vt ∈ V a time dependent vector field on Rn.• ϕt ∈ Diff , the flow defined by
∂tϕt = vt(ϕt) . (1)
Action of the group of diffeomorphism G0 (flow at time 1):
Π : G0 × C → C ,Π(ϕ,X )
.= ϕ.X
Right-invariant metric on G0: d(ϕ0,1, Id)2 = 12
∫ 10|vt |2V dt.
−→ Strong metric needed on V(Mumford and Michor: Vanishing Geodesic Distance on...)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Matching problems in a diffeomorphic framework
1 U a domain in Rn
2 V a Hilbert space of C 1 vector fields such that:
‖v‖1,∞ 6 C |v |V .
V is a Reproducing kernel Hilbert Space (RKHS): (pointwiseevaluation continuous)=⇒ Existence of a matrix function kV (kernel) defined on U × Usuch that:
〈v(x), a〉 = 〈kV (., x)a, v〉V .
Right invariant distance on G0
d(Id, ϕ)2 = infv∈L2([0,1],V )
∫ 10
|vt |2V dt ,
−→ geodesics on G0.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Matching problems in a diffeomorphic framework
1 U a domain in Rn
2 V a Hilbert space of C 1 vector fields such that:
‖v‖1,∞ 6 C |v |V .
V is a Reproducing kernel Hilbert Space (RKHS): (pointwiseevaluation continuous)=⇒ Existence of a matrix function kV (kernel) defined on U × Usuch that:
〈v(x), a〉 = 〈kV (., x)a, v〉V .
Right invariant distance on G0
d(Id, ϕ)2 = infv∈L2([0,1],V )
∫ 10
|vt |2V dt ,
−→ geodesics on G0.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Matching problems in a diffeomorphic framework
1 U a domain in Rn
2 V a Hilbert space of C 1 vector fields such that:
‖v‖1,∞ 6 C |v |V .
V is a Reproducing kernel Hilbert Space (RKHS): (pointwiseevaluation continuous)=⇒ Existence of a matrix function kV (kernel) defined on U × Usuch that:
〈v(x), a〉 = 〈kV (., x)a, v〉V .
Right invariant distance on G0
d(Id, ϕ)2 = infv∈L2([0,1],V )
∫ 10
|vt |2V dt ,
−→ geodesics on G0.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Variational formulation
Find the best deformation, minimize
J (ϕ) = infϕ∈GV
d(ϕ.A,B)2︸ ︷︷ ︸similarity measure
(2)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Variational formulation
Find the best deformation, minimize
������
������J (ϕ) = inf
ϕ∈GVd(ϕ.A,B)2︸ ︷︷ ︸
similarity measure
(2)
Tychonov regularization:
J (ϕ) = R(ϕ)︸ ︷︷ ︸Regularization
+1
2σ2d(ϕ.A,B)2︸ ︷︷ ︸
similarity measure
. (3)
Riemannian metric on GV :
R(ϕ) =1
2
∫ 10
|vt |2V dt (4)
is a right-invariant metric on GV .
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Optimization problem
Minimizing
J (v) = 12
∫ 10
|vt |2V dt +1
2σ2d(ϕ0,1.A,B)
2 .
In the case of landmarks:
J (ϕ) = 12
∫ 10
|vt |2V dt +1
2σ2
k∑i=1
‖ϕ(xi )− yi‖2 ,
In the case of images:
d(ϕ0,1.I0, Itarget)2 =
∫U
|I0 ◦ ϕ1,0 − Itarget |2dx .
Main issues for practical applications:
• choice of the metric (prior),• choice of the similarity measure.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Optimization problem
Minimizing
J (v) = 12
∫ 10
|vt |2V dt +1
2σ2d(ϕ0,1.A,B)
2 .
In the case of landmarks:
J (ϕ) = 12
∫ 10
|vt |2V dt +1
2σ2
k∑i=1
‖ϕ(xi )− yi‖2 ,
In the case of images:
d(ϕ0,1.I0, Itarget)2 =
∫U
|I0 ◦ ϕ1,0 − Itarget |2dx .
Main issues for practical applications:
• choice of the metric (prior),• choice of the similarity measure.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Why does the Riemannian framework matter?
Generalizations of statistical tools in Euclidean space:
• Distance often given by a Riemannian metric.• Straight lines → geodesic defined by
Variational definition: arg minc(t)
∫ 10
‖ċ‖2c(t) dt = 0 ,
Equivalent (local) definition: ∇ċ ċ = c̈ + Γ(c)(ċ , ċ) = 0 .
• Average → Fréchet/Karcher mean.
Variational definition: arg min{x → E [d2(x , y)]dµ(y)}Critical point definition: E [∇x d2(x , y)]dµ(y)] = 0 .
• PCA → Tangent PCA or PGA.• Geodesic regression, cubic regression...(variational or
algebraic)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Karcher mean on 3D images
Init. guesses
1 iteration
2 iterations
3 iterationsA1i A
2i A
3i A
4i
Figure – Average image estimates Ami , m ∈ {1, · · · , 4} after i =0, 1, 2and 3 iterations.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Interpolation, Extrapolation
Figure – Geodesic regression (MICCAI 2011)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Interpolation, Extrapolation
Figure – Extrapolation of happiness
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learningWhat metric to choose?
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Choosing the right-invariant metricRight-invariant metric: Eulerian fluid dynamic viewpoint onregularization.Space V of vector fields is defined equivalently by
• its kernel K such as Gaussian kernel,• its differential operator, for instance (Id−σ∆)n for Sobolev
spaces.
The norm on V is simply
‖v‖2V =∫
Ω
〈v(x), (Lv)(x)〉 dx =∫
Ω
(L1/2v)2(x) dx .
Scale parameter important!
kσ(x , y) = e− ‖x−y‖
2
σ2 kernel/operator (Id−σ∆)n (5)
• σ small: good matching but non regular deformations andmore local minima.
• σ large: poor matching but regular deformations and moreglobal minima.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Choosing the right-invariant metricRight-invariant metric: Eulerian fluid dynamic viewpoint onregularization.Space V of vector fields is defined equivalently by
• its kernel K such as Gaussian kernel,• its differential operator, for instance (Id−σ∆)n for Sobolev
spaces.
The norm on V is simply
‖v‖2V =∫
Ω
〈v(x), (Lv)(x)〉 dx =∫
Ω
(L1/2v)2(x) dx .
Scale parameter important!
kσ(x , y) = e− ‖x−y‖
2
σ2 kernel/operator (Id−σ∆)n (5)
• σ small: good matching but non regular deformations andmore local minima.
• σ large: poor matching but regular deformations and moreglobal minima.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Choosing the right-invariant metricRight-invariant metric: Eulerian fluid dynamic viewpoint onregularization.Space V of vector fields is defined equivalently by
• its kernel K such as Gaussian kernel,• its differential operator, for instance (Id−σ∆)n for Sobolev
spaces.
The norm on V is simply
‖v‖2V =∫
Ω
〈v(x), (Lv)(x)〉 dx =∫
Ω
(L1/2v)2(x) dx .
Scale parameter important!
kσ(x , y) = e− ‖x−y‖
2
σ2 kernel/operator (Id−σ∆)n (5)
• σ small: good matching but non regular deformations andmore local minima.
• σ large: poor matching but regular deformations and moreglobal minima.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Sum of kernels and multiscaleChoice of mixture of Gaussian kernels: (Risser, Vialard et al. 2011)
K (x , y) =n∑
i=1
αi e− ‖x−y‖
2
σ2i (6)
Figure – Left to right: Small scale, large scale and multi scale.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Sum of kernels and multiscaleChoice of mixture of Gaussian kernels: (Risser, Vialard et al. 2011)
K (x , y) =n∑
i=1
αi e− ‖x−y‖
2
σ2i (6)
Figure – Left to right: Small scale, large scale and multi scale.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Decomposition over different scales
Try to disentangle contributions at each scale: (Bruveris, Risser,Vialard, 2012, Siam MMS) using semi-direct product of groups.Consider Gσ1 ,Gσ2 two diffeomorphism groups at different scalesassociated with Vσ1 and Vσ2 .
Semi-direct product of groups Gσ1 n Gσ2 .
Non-linear extension of the infimal convolution of norms:
‖v‖2 = min(v1,v2)∈V1×V2
{‖v1‖2V1 + ‖v2‖
2V2
∣∣∣ v = v1 + v2} . (7)Non-linear extension −→ semi-direct product of groups.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
From Eulerian to Lagrangian viewpoints
Spatial correlation of the deformation: need for local deformabilityon the tissues.Toward a more Lagrangian point of view.
How to introduce spatially varying metric?
Using kernels: χi being a partition of unity of the domain.
K =n∑
i=1
χi Kiχi , . (8)
This kernel is associated to the following variational interpretation:
‖v‖2 = min(v1,...,vn)∈V1×...×Vn
{n∑
i=1
‖vi‖2Vi∣∣∣ n∑
i=1
χi vi = v
}. (9)
−→ possibility to introduce soft-symmetries...
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
From Eulerian to Lagrangian viewpoints
Spatial correlation of the deformation: need for local deformabilityon the tissues.Toward a more Lagrangian point of view.
How to introduce spatially varying metric?
Using kernels: χi being a partition of unity of the domain.
K =n∑
i=1
χi Kiχi , . (8)
This kernel is associated to the following variational interpretation:
‖v‖2 = min(v1,...,vn)∈V1×...×Vn
{n∑
i=1
‖vi‖2Vi∣∣∣ n∑
i=1
χi vi = v
}. (9)
−→ possibility to introduce soft-symmetries...
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Left-invariant metrics
Miccai 2013, Marsden’s Fields volume, Schmah, Risser, VialardChange of point of view: choose body-coordinates and convectivevelocity:
J (ϕ) = 12
∫ 10
‖v(t)‖2V dt + E (ϕ(1) · I , J), (10)
under the convective velocity constraint:
∂tϕ(t) = dϕ(t) · v(t) , (11)
where dϕ(t) is the tangent map of ϕ(t).
• More natural interpretation of spatially varying metrics.• Left action + left invariant metric =⇒ no induced
Riemannian metric.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Left-invariant metrics
Miccai 2013, Marsden’s Fields volume, Schmah, Risser, VialardChange of point of view: choose body-coordinates and convectivevelocity:
J (ϕ) = 12
∫ 10
‖v(t)‖2V dt + E (ϕ(1) · I , J), (10)
under the convective velocity constraint:
∂tϕ(t) = dϕ(t) · v(t) , (11)
where dϕ(t) is the tangent map of ϕ(t).
• More natural interpretation of spatially varying metrics.• Left action + left invariant metric =⇒ no induced
Riemannian metric.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Difference with LDDMMThe path look different:
Figure – The green curves Right-LDM geodesic path, The blue curvesLeft-LDM geodesic path.
LDDMM LIDM
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
What’s next
Left-invariance is more Lagrangian but the metric is fixed as in theEulerian situation!
• On a template, learning the metric.
Motivation:
• Better matching results: i.e better regularization or matching.• Better matching quality for organs with (segmented) tumors.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Metric learning
Figure – Given a collection of shape and a template, learn the metric.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Metric learning: High-dimensional inverseproblem
(Miccai 2014: Vialard, Risser)
• (In)n=1,...,N be a population of N images.• T be a template (Karcher mean for instance).
Registering the template T onto the image In consists inminimizing:
JIn,K (v) =1
2
∫ 10
‖v(t)‖2V dt + d(T ◦ ϕ(1)−1, In) , (12)
where∂tϕ(t) = v(t) ◦ ϕ(t) .
Equivalent to minimize
JIn,K (v) =1
2
∫ 10
〈P(t)∇I (t),K?(P(t)∇I (t)〉 dt+d(T◦ϕ(1)−1, In) ,
(13)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Metric learning: High-dimensional inverseproblem
(Miccai 2014: Vialard, Risser)
• (In)n=1,...,N be a population of N images.• T be a template (Karcher mean for instance).
Registering the template T onto the image In consists inminimizing:
JIn,K (v) =1
2
∫ 10
‖v(t)‖2V dt + d(T ◦ ϕ(1)−1, In) , (12)
where∂tϕ(t) = v(t) ◦ ϕ(t) .
Equivalent to minimize
JIn,K (v) =1
2
∫ 10
〈P(t)∇I (t),K?(P(t)∇I (t)〉 dt+d(T◦ϕ(1)−1, In) ,
(13)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Metric learning: High-dimensional inverseproblem
(Miccai 2014: Vialard, Risser)
• (In)n=1,...,N be a population of N images.• T be a template (Karcher mean for instance).
Registering the template T onto the image In consists inminimizing:
JIn,K (v) =1
2
∫ 10
‖v(t)‖2V dt + d(T ◦ ϕ(1)−1, In) , (12)
where∂tϕ(t) = v(t) ◦ ϕ(t) .
Equivalent to minimize
JIn,K (v) =1
2
∫ 10
〈P(t)∇I (t),K?(P(t)∇I (t)〉 dt+d(T◦ϕ(1)−1, In) ,
(13)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Optimize over K? Ill posed!• Incorporate the smoothness constraint by defining
K = {K̂ MK̂ |M SDP operator on L2(Rd ,Rd )} , (14)
M symmetric positive definite matrix.
• Regularization on M. Prior for M close to Id. Minimize
F(M) = β2
d2S++ (M, Id) +1
N
N∑n=1
minvJIn (v ,M) , (15)
where d2 can be chosen as
• Affine invariant metric (Pennec et al.)g1 = Tr(M
−1(δM)M−1(δM)).
• (Modified) Wasserstein metric.
Problem
Problem: matrix M is huge: (dn)2 where d = 2, 3 dimension andn number of voxels.Computing the logarithm is costly.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Optimize over K? Ill posed!• Incorporate the smoothness constraint by defining
K = {K̂ MK̂ |M SDP operator on L2(Rd ,Rd )} , (14)
M symmetric positive definite matrix.• Regularization on M. Prior for M close to Id. Minimize
F(M) = β2
d2S++ (M, Id) +1
N
N∑n=1
minvJIn (v ,M) , (15)
where d2 can be chosen as
• Affine invariant metric (Pennec et al.)g1 = Tr(M
−1(δM)M−1(δM)).
• (Modified) Wasserstein metric.
Problem
Problem: matrix M is huge: (dn)2 where d = 2, 3 dimension andn number of voxels.Computing the logarithm is costly.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Wasserstein metric
Pros: Easy to computeCons: Non complete metric.
Trick
The map N 7→ NNT is a Riemannian submersion from Mn(R)equipped with the Frobenius norm to the space of SDP matricesequipped with the Wasserstein metric
Encode the symmetric matrix as NNT and perform theoptimization on N. The regularization reads 12‖N − Id‖
2
The gradient is
∇L2F(N) = β(N − Id)− (16)
1
2N
N∑n=1
∫ 10
(K̂ ? Pn(t))⊗ (NK̂ ? Pn(t)) + (NK̂ ? Pn(t))⊗ (K̂ ? Pn(t))dt ,
(17)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Wasserstein metric
Pros: Easy to computeCons: Non complete metric.
Trick
The map N 7→ NNT is a Riemannian submersion from Mn(R)equipped with the Frobenius norm to the space of SDP matricesequipped with the Wasserstein metric
Encode the symmetric matrix as NNT and perform theoptimization on N. The regularization reads 12‖N − Id‖
2
The gradient is
∇L2F(N) = β(N − Id)− (16)
1
2N
N∑n=1
∫ 10
(K̂ ? Pn(t))⊗ (NK̂ ? Pn(t)) + (NK̂ ? Pn(t))⊗ (K̂ ? Pn(t))dt ,
(17)
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Reducing the problem dimension
IdeaLearn at large scales and not at fine scale.
Introduce:
K = {K̂ MΠK̂ + K̂ (Id − Π)K̂ |M SDP operator on L2(Rd ,Rd )} .(18)
with Π an orthogonal projection on a finite dimensionalparametrization of vector fields: use of splines.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Experiments• 40 subjects of the LONI Probabilistic Brain Atlas (LPBA40).• All 3D images were affinely aligned to subject 5 using ANTS.
Table – Reference results
No Reg SyN Kfine Kref
TO 0.665 0.750 0.732 0.712
DetJMax 1 3.17 4.65 1.66DetJMin 1 0.047 0.46 0.67DetJStd 0 0.17 0.11 0.063
Table – Average results.
DiagM GridM1 GridM2 K20 K30
TO 0.711 0.710 0.704 0.710 0.704
DetJMax 1.66 1.61 1.41 1.62 1.50DetJMin 0.68 0.70 0.67 0.73 0.66DetJStd 0.062 0.059 0.049 0.056 0.063
For a given quality of overlap, better smoothness of thedeformations.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Experiments• 40 subjects of the LONI Probabilistic Brain Atlas (LPBA40).• All 3D images were affinely aligned to subject 5 using ANTS.
Table – Reference results
No Reg SyN Kfine Kref
TO 0.665 0.750 0.732 0.712
DetJMax 1 3.17 4.65 1.66DetJMin 1 0.047 0.46 0.67DetJStd 0 0.17 0.11 0.063
Table – Average results.
DiagM GridM1 GridM2 K20 K30
TO 0.711 0.710 0.704 0.710 0.704
DetJMax 1.66 1.61 1.41 1.62 1.50DetJMin 0.68 0.70 0.67 0.73 0.66DetJStd 0.062 0.059 0.049 0.056 0.063
For a given quality of overlap, better smoothness of thedeformations.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Main issues
• The metric is fixed in Eulerian coordinates.• The metric is template based.
Make the method adaptive to any pairs of images on a simplermodel.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
SVF model: A simple model
Based on Metric learning for image registration, CVPR 2019,Niethammer, Kwitt, Vialard.Let v(x) be a vector field. Find a v minimizer of
1
2‖v‖2V + Sim(I ◦ ϕ−11 , J) (19)
∂tϕt = v(ϕt) . (20)
Equivalent momentum formulation: Find m ∈ L2(Ω,Rd ) ⊂ V ∗such that
1
2〈m,K ?m〉+ Sim(I ◦ ϕ−11 , J) (21)
s.t.∂tϕ−1(t, x) + Dϕ−1(t, x)(K ?m(t, x)) = 0 . (22)
Numerical discretization: Central differences in space and 20timesteps in time of RK4.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
SVF model: A simple model
Based on Metric learning for image registration, CVPR 2019,Niethammer, Kwitt, Vialard.Let v(x) be a vector field. Find a v minimizer of
1
2‖v‖2V + Sim(I ◦ ϕ−11 , J) (19)
∂tϕt = v(ϕt) . (20)
Equivalent momentum formulation: Find m ∈ L2(Ω,Rd ) ⊂ V ∗such that
1
2〈m,K ?m〉+ Sim(I ◦ ϕ−11 , J) (21)
s.t.∂tϕ−1(t, x) + Dϕ−1(t, x)(K ?m(t, x)) = 0 . (22)
Numerical discretization: Central differences in space and 20timesteps in time of RK4.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Parametrization of the metricFix a collection of scales σ0 < . . . < σN−1 and set
Gi (x) = e−|x|2/σ2i .
v0(x)def.= (K (w) ?m0)(x)
def.=
N−1∑i=0
√wi (x)
∫y
Gi (|x − y |)√
wi (y)m0(y) dy , (23)
Problem: wi should be sufficiently smooth to guaranteediffeomorphisms.Introduce pre-weights ωi (x) and fix Kσ, with σ small:
Kσ ? ωi = wi
and ∑i
wi (x) = 1 .
Learning the metric is still ill-posed:
ÔMT(w) =
∣∣∣∣log σN−1σ0∣∣∣∣−r N−1∑
i=0
wi
∣∣∣∣log σN−1σi∣∣∣∣r (24)
Is 0 for (wi ) = (0, . . . , 0, 1).
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Objective function
Optimize momentum + metric.
Obj0,1(m, ω) = argminm0
λ〈m0, v0〉 + Sim[I0 ◦ Φ−1(1), I1] +
λOMT
∫ÔMT(w(x)) dx +
λTV
√√√√N−1∑l=0
(∫γ(‖∇I0(x)‖)‖∇ωl (x)‖2 dx
)2, (25)
where γ(x) ∈ R+ is an edge indicator function
γ(‖∇I‖) = (1 + α‖∇I‖)−1, α > 0 .
Then, minimize ∑i,j
Obj0,1(mi,j , ωi ) , (26)
where ωi = fθ(Ii ).
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Parametrize and learn the pre-weights ωi
The pre-weights are parametrized by a 2-layers net:
(ωi )i=1,...,N = ShallowNet(I ) .
Input: current image, Output: pre-weights.
ShallowNet = conv(d , n1) → BatchNorm → lReLU →conv(n1,N) → BatchNorm → weighted-linear-softmax
σw (z)j =clamp0,1(wj + zj − z)∑N−1
i=0 clamp0,1(wi + zi − z), (27)
The weights wj are reasonably initialized: wi = σ2i /(∑N−1
j=0 σ2j )
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Optimization
Shared parameters: ShallowNetwork parameters,Individual parameters: Momentum for each pair.
1 (1) initialize with reasonable weights and optimize overmomentums,
2 (2) Jointly optimize on the shared and individual parameters:use SGD with (Nesterov) momentum, different batch size in2d/3d, 50 epochs in 2d, less in 3D.
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Experiments on synthetic data1) Generate concentric circular regions with random radii and
associate different multi-Gaussian weights to these regions.We associate a fixed multi-Gaussian weight to thebackground.
2) Randomly create vector momenta at the borders of theconcentric circles.
3) Add noise (for texture) and compute forward model, toobtain source image, similar for target image.
0
1
2
3
disp
erro
r (es
t-GT)
[pix
el]
glob
alt=
0.01
;o=5
_l
t=0.
01;o
=15_
l
t=0.
01;o
=25_
l
t=0.
01;o
=50_
l
t=0.
01;o
=75_
l
t=0.
01;o
=100
_l
t=0.
10;o
=5_l
t=0.
10;o
=15_
l
t=0.
10;o
=25_
l
t=0.
10;o
=50_
l
t=0.
10;o
=75_
l
t=0.
10;o
=100
_l
t=0.
25;o
=5_l
t=0.
25;o
=15_
l
t=0.
25;o
=25_
l
t=0.
25;o
=50_
l
t=0.
25;o
=75_
l
t=0.
25;o
=100
_l0.0
0.5
1.0
1.5
2.0
disp
erro
r (es
t-GT)
[pix
el]
Figure – Displacement error (in pixel) with respect to the ground truthfor various values of the total variation penalty, λTV (t) and the OMTpenalty, λOMT (o). Results for the interior (top) and the exterior(bottom) rings show subpixel registration accuracy for all local metricoptimization results (* l). Overall, local metric optimizationsubstantially improves registrations over the results obtained via initialglobal multi-Gaussian regularization (global).
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Experiments on synthetic dataSource image Target image
Warped source Deformation grid Std.dev.
λOMT = 150.165
0.170
0.175
0.180
0.185
0.190
0.195
λOMT = 500.170
0.175
0.180
0.185
0.190
0.195
λOMT = 100 0.1650.170
0.175
0.180
0.185
0.190
0.195
Figure – λTV = 0.1. Overall variance is similar but the true weights arenot recovered: weights on the outer ring [0.05, 0.55, 0.3, 0.1]
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Experimentsw0(σ = 0.01) w1(σ = 0.05) w2(σ = 0.10) w3(σ = 0.20)
λOMT = 15
λOMT = 50
λOMT = 100
Figure
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
On 2D real data: LPBA40
Source image Target image
Warped source Deformation grid Std. dev.
λOMT = 15 0.1910.192
0.193
0.194
0.195
0.196
0.197
λOMT = 50 0.1920.193
0.194
0.195
0.196
0.197
0.198
λOMT = 1000.194
0.195
0.196
0.197
0.198
0.199
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Performance on 3D data: CUMC12
Training on different dataset:
Trained on different dataset, test on CUMC12
Training 132 image pairs on CUMC12, 90 image pairs onMGH10, 150 image pairs on IBSR18.
Method mean std 1% 5% 50% 95% 99% p MW-stat sig?FLIRT 0.394 0.031 0.334 0.345 0.396 0.442 0.463
Metric learning fordiffeomorphic image
registration.
François-XavierVialard
Introduction todiffeomorphisms groupand Riemannian tools
Choice of the metric
Spatially dependentmetrics
Metric learning
SVF metric learning
Conclusion
Summary
Adaptive metric learning in SVF.
Avoid end to end training for preserving diffeomorphicproperties.
Diffeomorphic guarantees at test time (no guarantee for DLmethods: VoxelMorph).
Perspectives
Combine it with momentum prediction (QuickSilver like).
Use it in LDDMM.
Incorporate richer deformations descriptors.
Paper to appear: Metric learning for image registration, CVPR2019, Niethammer, Kwitt, Vialard.
Main TalkIntroduction to diffeomorphisms group and Riemannian toolsChoice of the metricSpatially dependent metricsMetric learningSVF metric learning
anm0: anm1: