Upload
arleen-todd
View
216
Download
0
Embed Size (px)
Citation preview
FEATURE-BASED ALIGNMENTOF VOLUMETRIC MULTI-MODAL IMAGES
Matthew Toews, Lilla Zöllei, William Wells III June 29, 2013 IPMI
Challenges: Arbitrary subjects, anatomies (brain, body, …), modalities (MR, CT, …), pathology, lack of one-to-one homology, unknown initialization, DICOM errors, …
ROBUST IMAGE ALIGNMENT
Thoracic CTInfant Brain MR
TBI, MR
Brain MR, CT(tumor)
2
Robust clinical usage Initialize registration, segmentation routines
Large-scale data mining, e.g. Google style
APPLICATIONS
Thoracic CTInfant Brain MR
TBI, MR
Brain MR, CT(tumor)
3
FEATURE-BASED ALIGNMENT METHOD
Alignment via 3D scale-invariant feature correspondences
Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.
4
FEATURE-BASED ALIGNMENT METHOD
Strengths Robust: lack of one-to-one homology, disease, resection,
Globally optimal: no ‘capture radius’, initialization
Efficient: Memory, computation
Useful: Alignment, disease classification, prediction
Weaknesses Does not align different modalities Requires pre-aligned training data
Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.
5
CONTRIBUTIONS
Inverted local feature correspondence Extend scale-invariant feature representation to
multi-modal alignment
Group-wise feature-based alignment Remove requirement for pre-aligned training
data Multiple modalities
6
OVERVIEW
Scale-invariant feature representation
Inverted feature correspondence
Group-wise feature-based alignment
OVERVIEW
Scale-invariant feature representation
Inverted feature correspondence
Group-wise feature-based alignment
SCALE-INVARIANT FEATURES
Distinctive image patches Image-to-image
correspondence SIFT method: computer vision
Invariant to scaling, rotation, translation, illumination changes
Fast, efficient, robust Large-scale image search
Distinctive Image Features from Scale-Invariant KeypointsD. G. Lowe, IJCV, 2004.
9
3D Geometry S = {X, σ, Θ} Location X = (x, y, z) Scale σ Orientation
(axis: 3 unit vectors )
Appearance I Intensity descriptor
SCALE-INVARIANT FEATURES IN 3D
Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.
S I
)ˆ,ˆ,ˆ( 321
10
Difference-of-Gaussian scale-space extrema.
IDENTIFYING LOCATION X, SCALE σ
d
xdI ),()(xI
x
),()(),( xGxIxI
Feature locations, scales:blob-like image patterns
d
xdIx
),(localmax
,
iiX ,
11
Dominant local image gradient directions)( IH 3D gradient orientation histogram
I
I
Vote location (upon unit sphere)
IVote magnitude
ASSIGNING ORIENTATION Θ
Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.
)(argmaxˆˆ
1 IH
))ˆ(ˆ( argmaxˆ11
ˆ2
IH 213
ˆˆˆ
)(XII )( IH
12
Encode local image content For image correspondence / matching
Gradient orientation histogram (GoH) Quantization: 8 location x 8 orientation
bins 113 voxels → 64 bins (small size)
Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.
FEATURE DESCRIPTOR I
IGoHNormalized
Image PatchI
)(XI
13
OVERVIEW
Scale-invariant feature representation
Inverted feature correspondence
Group-wise feature-based alignment
MATCHING ACROSS MODALITIES
Joint Intensity relationship Globally multi-modal Locally linear
MP-RAGE
T2
negative localcorrelation
positive localcorrelation
Non-rigid registration of multi-modal images using both mutual information and cross-correlation.Medical Image Analysis, 2008. A. Andronache, M.V. Siebenthal, G. Szekely, P. Cattin.
15
MATCHING ACROSS MODALITIES
Positive local correlation Conventional correspondence methods,
descriptor matching
MP-RAGE
T2positivecorrelation
16
MATCHING ACROSS MODALITIES
Negative local correlation Conventional correspondence fails Inverted local gradient, orientations ,
descriptor
MP-RAGE
T2
negativecorrelation
21ˆ,ˆ
17
MATCHING ACROSS MODALITIES
Inverted correspondence Rotate orientation, descriptor elements by -π
about Correspondence successful
MP-RAGE
T2
negativecorrelation
3̂
18
MATCHING ACROSS MODALITIES
MP-RAGE, T2, intra-subject No conventional correspondences in GM / WM 22 inverted correspondences within GM / WM
MP-RAGE
T2
19
MATCHING ACROSS MODALITIES
Infant T1 MR: newborn ↔ 2 years old GM / WM contrast inversion due to mylenation No conventional correspondences in GM / WM 4 inverted correspondences within GM / WM
Unbiased average age-appropriate atlases for pediatric studiesNeuroImage 2011. V.S. Fonov, A.C. Evans, D.L. Collins et al.
20
OVERVIEW
Scale-invariant feature representation
Inverted feature correspondence
Group-wise feature-based alignment
GROUP-WISE ALIGNMENT
Automatically align a set of subject images Arbitrary initialization, modalities
22
GROUP-WISE ALIGNMENT: MODEL
ji
iijiji TSIpTp,
)|,(})({
}{ iT
},{ ijij SI
Transform set: image i to atlas (similarity transform)
Feature descriptor, geometry set: image i, feature j
Bayes rule
Conditional feature independence
Marginalization over F
}{ ,lkfF Latent model feature set, feature k,conventional & inverted modes l={0,1}
}){|},({})({}),{|}({ iijijiijiji TSIpTpSITp
ji
lklk
ilkijiji fpTfSIpTp,
,,
, )(),|,(})({
Input features (Iij,Sij) are - Conditionally independent - Identically distributed according to a Gaussian mixture model
23
GAUSSIAN MIXTURE MODEL (GMM)
lk
lklk fpTfSIpTSIp,
,, )(),|,()|,(
)(),|,( 0,30,3 fpTfSIp)(),|,( 0,10,1 fpTfSIp
)(),|,( 0,20,2 fpTfSIp
)(),|,( 0,00,0 fpTfSIp
),( SI
Background24
GMM: UNRECOGNIZED INPUT FEATURE
lk
lklk fpTfSIpTSIp,
,, )(),|,()|,(
)(),|,( 0,30,3 fpTfSIp)(),|,( 0,10,1 fpTfSIp
)(),|,( 0,20,2 fpTfSIp
),( SI
),( SI
)(),|,( 0,00,0 fpTfSIp
Background25
)(),|,( 0,30,3 fpTfSIp
GMM: CONVENTIONAL INPUT FEATURE
lk
lklk fpTfSIpTSIp,
,, )(),|,()|,(
)(),|,( 0,30,3 fpTfSIp)(),|,( 0,10,1 fpTfSIp
)(),|,( 0,20,2 fpTfSIp
),( SI
),( SI
)(),|,( 0,00,0 fpTfSIp
Background26
GMM: INVERTED INPUT FEATURE
lk
lklk fpTfSIpTSIp,
,, )(),|,()|,(
)(),|,( 0,30,3 fpTfSIp)(),|,( 0,10,1 fpTfSIp
)(),|,( 0,20,2 fpTfSIp
),( SI
),( SI
)(),|,( 0,00,0 fpTfSIp
Background
)(),|,( 1,21,2 fpTfSIp
27
LIKELIHOOD: APPEARANCE DESCRIPTOR
lk
lkilkijijiijij fpTfSIpTSIp,
,, )(),|,()|,(
),|()|(),|,( ,,, ilkijlkijilkijij TfSpfIpTfSIp Conditional independence
Descriptor:Isotropic Gaussian density over descriptor elements.
Note: descriptor conditionally independent of alignment Ti,- Fast Matching, no search over Ti -
28
),|(),|(),|()|(),|,( ,,,,, ilkijilkijilkijlkijilkijij TfpTfpTfXpfIpTfSIp
LIKELIHOOD: GEOMETRY
Location: Isotropic Gaussian
lk
lkilkijijiijij fpTfSIpTSIp,
,, )(),|,()|,(
Scale:Gaussian in log σij
Orientation:Isotropic GaussianApproximate Von Mises
29
PRIOR
Latent feature probability: Discrete
lk
lkilkijijiijij fpTfSIpTSIp,
,, )(),|,()|,(
0, lkf
1, lkf
Conventional appearance mode
Inverted appearance mode
30
GROUP-WISE ALIGNMENT: ALGORITHM
Inputs:Volumetric images
Outputs:Alignment solutions: {Ti}
Feature-based model: {fk,l}, p(Iij, Sij| fk,l,Ti), p(fk,l)
31
GROUP-WISE ALIGNMENT: ALGORITHM
1) Feature extraction2) Initialization
Approximate {Ti}
3) Model LearningFixed {Ti}, vary {fk,l}, p(Iij, Sij| fk,l,Ti), p(fk,l)
Mixture model; density, probability parameter estimation.
4) Alignment / Model FittingFixed {fk,l}, p(Iij, Sij| fk,l,Ti), p(fk,l), vary {Ti}
Subject-to-model alignment.
Iterate between 3) & 4) until convergence, i.e. {Ti} no longer changes.
32
1) FEATURE EXTRACTION
Features extracted once from individual images
Note: algorithms use feature data only ~100X data reduction compared to original
image volumes. ~25 seconds for 2563-voxel image, standard PC. ~2K features per brain image
33
2) INITIALIZATION
Approximate image alignment Nearest neighbor descriptor matching Hough transform, similarity
Note: some initial misalignment OK A small subset of image should be aligned Misaligned features sets have negligible impact
34
3) MODEL ESTIMATION
Mixture modeling Estimate set {fk,l}, parameters of p(Iij, Sij| fk,l,Ti), p(fk,l) Robust feature clustering across subjects
Similar to mean-shift algorithm
Note Model conventional appearance Same structure, two distinct latent features, e.g.:
0,1 lkf0,2 lkf
Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.
35
4) MODEL FITTING
Maximum a-posteriori estimation Maximize Ti individually
Conditional independencei
ijijiijiji SITpSITp }),{|(}),{|}({
}),{|( argmax ijijiT
iMAP SITpTi
36
4) MODEL FITTING
Maximum a-posteriori estimation Maximize Ti individually
Two approaches Conventional
Multi-Modal (conventional & inverted modes) Assume
Conditional independencei
ijijiijiji SITpSITp }),{|(}),{|}({
}),{|( argmax ijijiT
iMAP SITpTi
)()( 0,1, lklk fpfp
0)( 1, lkfp
37
EXPERIMENTS
Group-wise Alignment: RIRE data set Modalities: T1, T2, PD, MP-RAGE, CT Brain: 9 subjects, 39 images All subjects exhibit brain tumors
Difficult problem: subject abnormality, no prior information used regarding modalities, initialization.
Compare conventional vs. multi-modal fitting
Comparison and evaluation of retrospective intermodality brain image registration techniquesJournal of Computer Assisted Tomography, 1997. J.B. West, J.M. Fitzpatrick et al.
38
RESULTS: ALIGNMENT
39
RESULTS
Conventional alignment: 3 failure cases (all CT)
Multi-Modal alignment: success
40
Failure case
Success
Failure
RESULTS
Model feature examples
41
DISCUSSION: INVERTED CORRESPONDENCE
Useful for fitting/matching between modalities Less useful once model has been learned May be more prone to false correspondences
Analogous to mutual information Useful when prior information is weak
42
A marginalized MAP approach and EM optimization for pair-wise registrationIPMI 2007. L. Zollei, M. Jenkinson, S. Timoner, W.M. Wells III
DISCUSSION: GROUP-WISE ALIGNMENT
Alignment of difficult multi-modal data Unknown initialization Also effective for infant MR, torso CT, lung CT
Fast 22 minutes (vs. 10-20 hours for group-wise
registration) Deformable alignment?
Global similarity Ti + local linear deformations about correspondences.
43
ACKNOWLEDGEMENTS
NIH grants: P41-EB-015902
P41-RR-013218R00 HD061485-03P41-EB-015898P41-RR-019703
44