Computational Radiology LaboratoryHarvard Medical Schoolwww.crl.med.harvard.edu
Children’s Hospital Department of Radiology Boston Massachusetts
A Survey of Validation Techniques for Image Segmentation and Registration, with a focus on the STAPLE algorithm
Simon K. Warfield, Ph.D.
Associate Professor of Radiology
Harvard Medical School
Computational Radiology Laboratory. Slide 2
Outline
• Validation of image segmentation– Overview of approaches– STAPLE
• Validation of image registration
• STAPLE algorithm available as open source software from:– http://www.nitrc.org/projects/staple– http://crl.med.harvard.edu/
Computational Radiology Laboratory. Slide 3
Segmentation• Goal: identify or label structures
present in the image.• Many methods:
– Interactive or manual delineation,– Supervised approaches with user
initialization,– Alignment with a template,– Statistical pattern recognition.
• Applications:– Quantitative measurement of
volume, shape or location of structures,
– Provides boundary for visualization by surface rendering.
Newborn MRI Segmentation.
Computational Radiology Laboratory. Slide 4
Validation of Image Segmentation• Spectrum of accuracy versus realism in
reference standard.• Digital phantoms.
– Ground truth known accurately.– Not so realistic.
• Acquisitions and careful segmentation.– Some uncertainty in ground truth.– More realistic.
• Autopsy/histopathology.– Addresses pathology directly; resolution.
• Clinical data ?– Hard to know ground truth.– Most realistic model.
Computational Radiology Laboratory. Slide 5
Validation of Image Segmentation
• Comparison to digital and physical phantoms:– Excellent for testing the anatomy, noise and
artifact which is modeled.– Typically lacks range of normal or
pathological variability encountered in practice.
MRI of brain phantom from Styner et al. IEEE TMI 2000
Computational Radiology Laboratory. Slide 6
Comparison To Higher Resolution
MRI Photograph MRI
Provided by Peter Ratiu and Florin Talos.
Computational Radiology Laboratory. Slide 7
Comparison To Higher Resolution
Photograph MRI Photograph Microscopy
Provided by Peter Ratiu and Florin Talos.
Computational Radiology Laboratory. Slide 8
Comparison to Autopsy Data
• Neonate gyrification index– Ratio of length of cortical boundary to length
of smooth contour enclosing brain surface
Computational Radiology Laboratory. Slide 9
Staging
Stage 3 Stage 5
Stage 4 Stage 6
Stage 3: at 28 w GAshallow indentations of inf. frontal and sup. Temp. gyrus(1 infant at 30.6 w GA, normal range: 28.6 ± 0.5 w GA)
Stage 4: at 30 w GA2 indentations divide front. lobe into 3 areas, sup. temp.gyrus clearly detectable(3 infants, 30.6 w GA ± 0.4 w, normal range: 29.9 ± 0.3 w GA)
Stage 5: at 32 w GAfrontal lobe clearly divided into three parts: sup., middle and inf. Frontal gyrus(4 infants, 32.1 w GA ± 0.7 w,normal range: 31.6 ± 0.6 w GA)
Stage 6: at 34 w GAtemporal lobe clearly divided into
3 parts: sup., middle and inf. temporal gyrus(8 infants, 33.5 w GA ± 0.5 wnormal range: 33.8 ± 0.7 w GA)
“Assessment of cortical gyrus and sulcus formation using MR images in normalfetuses”, Abe S. et al., Prenatal Diagn 2003
Computational Radiology Laboratory. Slide 10
Neonate GI: MRI Vs AutopsyGyrification Index versus age in days
0
0.5
1
1.5
2
2.5
3
200 220 240 260 280 300 320 340
Post-conceptional age in days
GI
MRI Scan 2 MRI Scan 1 Armstrong
Computational Radiology Laboratory. Slide 11
GI Increase Is Proportional to Change in Age.
'change in GI' versus 'days of growth before final scan'
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
50 55 60 65 70 75 80 85 90
time interval between scans in days
ch
an
ge
of
GI
Change of Total Brain GI Linear (Change of Total Brain GI)
Computational Radiology Laboratory. Slide 12
GI Versus Qualitative StagingStaging versus GI
1
1.2
1.4
1.6
1.8
2
2.2
2.4
3 4 5 6 7 8 9
Staging Grade
To
tal
Bra
in G
I
MRI scan 1 MRI scan 2
Computational Radiology Laboratory. Slide 13
Neonate Gyrification
GI : interactive versus automatic segmentation.
y = 1.2241x + 0.4443
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
-1 0 1 2 3 4 5
GI - hand segmentation
GI -
au
tom
atic
seg
men
tati
on
Linear (line of equality)
Computational Radiology Laboratory. Slide 14
Validation of Image Segmentation• Comparison to expert performance; to other
algorithms.• Why compare to experts ?
– Experts are currently doing the segmentation tasks that we seek algorithms for.
– Surgical planning.– Neuroscience research.
• What is the appropriate measure for such comparisons ?
Computational Radiology Laboratory. Slide 15
Measures of Expert Performance• Repeated measures of volume
– Intra-class correlation coefficient
• Spatial overlap– Jaccard: Area of intersection over union.– Dice: increased weight of intersection.– Vote counting: majority rule, etc.
• Boundary measures– Hausdorff, 95% Hausdorff.
• Bland-Altman methodology:– Requires a reference standard.
• Measures of correct classification rate:– Sensitivity, specificity ( Pr(D=1|T=1), Pr(D=0|T=0) )– Positive predictive value and negative predictive value
(posterior probabilities Pr(T=1|D=1), Pr(T=0|D=0) )
Computational Radiology Laboratory. Slide 16
Validation of Image Segmentation
• STAPLE (Simultaneous Truth and Performance Level Estimation):– An algorithm for estimating performance
and ground truth from a collection of independent segmentations.
Computational Radiology Laboratory. Slide 17
STAPLE papers– Image segmentation with labels:
• Warfield, Zou, Wells ISBI 2002• Warfield, Zou, Wells MICCAI 2002.• Warfield, Zou, Wells, IEEE TMI 2004.• Commowick and Warfield IPMI 2009
– Image segmentation with boundaries:• Warfield, Zou, Wells MICCAI 2006.• Warfield, Zou, Wells PTRSA 2008.
– Diffusion data and vector fields:• Commowick and Warfield IEEE TMI 2009
Computational Radiology Laboratory. Slide 18
STAPLE: Estimation Problem
• Complete data density:• Binary ground truth Ti for each voxel i.
• Expert j makes segmentation decisions Dij.
• Expert performance characterized by sensitivity p and specificity q.
– We observe expert decisions D. If we knew ground truth T, we could construct maximum likelihood estimates for each expert’s sensitivity (true positive fraction) and specificity (true negative fraction):
)|( qp,TD,f
)|,(lnmaxargˆ,ˆ qp,TDqpqp,
f
Computational Radiology Laboratory. Slide 19
Expectation-Maximization• Since we don’t know ground truth T, treat T as a
random variable, and solve for the expert performance parameters that maximize:
• Parameter values θj=[pj qj]T that maximize the conditional expectation of the log-likelihood function are found by iterating two steps:– E-step: Estimate probability of hidden ground truth T given a
previous estimate of the expert quality parameters, and take expectation.
– M-step: Estimate expert performance parameters by comparing D to the current estimate of T.
€
Q(θ |θ (t−1)) = E ln f (D,T |θ) | D,θ (t−1)[ ]
Computational Radiology Laboratory. Slide 20
Probability Estimate of True Labels
Wsik f (T
is | D
i, k )
f (T
is) f (D
ij| T
is, k )
j
f (Ti s ) f (D
ij| T
i s , k )
j
s
Estimate probability of tissue class in reference standard:
Computational Radiology Laboratory. Slide 21
Binary Input: True Segmentation
, ,
,
( 1| , )
( | 1 ) ( 1)
( | , ) ( )
k k ki i
k kij i j j i
j
k kT ij i j j ii
j
k
k k
W f T
f D T p q f T
f D T p q f T
αα β
≡ =
= ==
∑
=+
∏∏
iD ,p q
: 1 : 0
: 0 : 1
( 1) (1 )
( 0) (1 )
( 1) : prior probability true label at voxel i is 1.
: conditional probability that true label is 1.
ij ij
ij ij
k k ki j jj D j D
k k ki j jj D j D
i
ki
f T p p
f T q q
f T
W
α
β= =
= =
= = −
= = −
=
∏ ∏∏ ∏
Computational Radiology Laboratory. Slide 22
Expert Performance Estimate
: 11
: 1 : 0
: 01
: 1 : 0
(1 )
(1 ) (1 )
ij
ij ij
ij
ij ij
kii Dk
j k ki ii D i D
kii Dk
j k ki ii D i D
Wp
W W
Wq
W W
p (sensitivity, true positive fraction) : ratio of expert identified class 1 to total class 1 in the image.
q (specificity, true negative fraction) : ratio of expert
identified class 0 to total class 0 in the image.
:1 ij
ksi
i D skjs s k
sii
W
W
Computational Radiology Laboratory. Slide 23
Newborn MRI Segmentation
Computational Radiology Laboratory. Slide 24
Newborn MRI Segmentation
Summary of segmentation quality (posterior probability Pr(T=t|D=t) ) for each tissue type for repeated manual segmentations.
Indicates limits of accuracy of interactive segmentation.
Computational Radiology Laboratory. Slide 25
Expert and Student Segmentations
Test image Expert consensus Student 1
Student 2 Student 3
Computational Radiology Laboratory. Slide 26
Phantom Segmentation
Image Expert Students Voting STAPLE
Image Expertsegmentation
Studentsegmentations
Computational Radiology Laboratory. Slide 27
STAPLE Summary• Key advantages of STAPLE:
– Estimates ``true’’ segmentation.– Assesses expert performance.
• Principled mechanism which enables:– Comparison of different experts.– Comparison of algorithm and experts.
• Extensions for the future:– Prior distribution or extended models for
expert performance characteristics.– Estimate bounds on parameters.
Computational Radiology Laboratory. Slide 28
Image registration
• A metric: measures similarity of images given an estimate of the transformation.
• Best metric depends on nature of the images.
• Alignment quality ultimately possible depends on model of transformation.
• The transformation is identified by solving an optimization problem.– Seek the transform parameters that
maximize the metric of image similarity
Computational Radiology Laboratory. Slide 29
Validation of Registration
• Compare transformations– Take some images, apply a transformation
to them.– Estimate the transform using registration– How well does the estimated transformation
match the applied transform?
• Check alignment of key image features– Fiducial alignment– Spatial overlap
• Segment structures, assess overlap after alignment.
Computational Radiology Laboratory. Slide 30
Intraoperative Nonrigid Registration
• Fast: it should not take more than 1 min to make the registration.
• Robust: the registration should work with poor quality image, artifacts, tumor...
• Physics based: we are not only concerned in the intensity matching, but also interested in recovering the physical (mechanical) deformation of the brain.
• Accurate: neuro-surgery needs a precise knowledge of the position of the structures.
• Archip et al. NeuroImage 2007
Computational Radiology Laboratory. Slide 31
Block Matching Algorithm
Divide a global optimization problem in many simple local ones
Highly parallelizable, as blocks can be matched independently.
Similarity measure: coefficient of correlation ]1:0[∈
Computational Radiology Laboratory. Slide 32
Block Matching Algorithm
Displacementestimates are noisy.
Computational Radiology Laboratory. Slide 33
Patient-specific Biomechanical Model
Pre-operativeimage
Automatic brain segmentation
Brain finite element model (linear elastic)
Computational Radiology Laboratory. Slide 34
Registration Validation
• Landmark matching assessment in six cases
• Parallel version runs in 35 seconds on a 10 dual 2GHz PC cluster – 7x7x7 block size– 11x11x25 window– 1x1x1 step– 50 000 blocks– 10 000 tetrahedra
Registration Error Evaluation Using Landmarks Correspondences
0
0,5
1
1,5
2
2,5
3
0 5 10 15
Displacement
Measured Error
Patient 1
Patient 2Patient 3
Patient 4Patient 5
Patient 6
• 60 landmarks:– Average error = 0.75mm– Maximum error = 2.5mm– Data voxel size 0.8x0.8x2.5 mm3
Computational Radiology Laboratory. Slide 35
Registration Validation
• 11 prospective consecutive cases,
• Alignment computed during the surgery.
• Estimate of the registration accuracy – 95% Hausdorff distance of the edges of the registered preoperative MRI and the intraoperative MRI.
Computational Radiology Laboratory. Slide 36
Automatic selection of fiducials
(1)Non-rigid alignment of preoperative MPRAGE.
Contours extracted from (1) with the Canny edge
detector
(2) Intraoperative whole brain SPGR at 0.5T
Contours extracted from (2) with the Canny edge
detector
95% Hausdorff metric
computed
Computational Radiology Laboratory. Slide 37
Alignment improvement
Tumor position Tumor pathology
Non-rigid registration – preop to intraop scans(95% Hausdorff distance)
Max Displacement
measured(mm)
Rigid registration
accuracy – preop to intraop
(mm)
Non-Rigid registration accuracy – preop to intraop(mm)
Ratio Rigid/Non-
Rigid
Case 1 right posterior frontal
oligoastrocytoma Grade II
10.68 5.95 1.90 3.13
Case 2 left posterior temporal
glioblastoma Grade IV 21.03 10.71 2.90 3.69
Case 3 left medial temporal
glioblastoma Grade IV 15.27 7.65 1.70 4.50
Case 4 left temporal anaplastic oligoastrocytoma Grade III
10.00 6.80 0.85 8.00
Case 5 right frontal oligoastrocytoma Grade II
9.87 5.10 1.27 4.01
Case 6 left frontal anaplastic astrocytoma Grade III
17.48 10.20 3.57 2.85
Case 7 right medial temporal
anaplastic astrocytoma Grade III
19.96 9.35 2.55 3.66
Case 8 right frontal oligoastrocytoma Grade II
17.44 8.33 1.19 7.00
Case 9 right frontotemporal
oligoastrocytoma Grade II
15.08 7.14 1.87 3.81
Case 10 right occipital anaplastic oligodendroglioma Grade III
9.48 5.95 1.44 4.13
Case 11 left frontotemporal
oligodendroglioma Grade II 10.74 4.76 0.85 5.60
AVG 14.27 7.44 1.82 4.58
Computational Radiology Laboratory. Slide 38
Visualization of aligned data
• Matched preoperative fMRI and DT-MRI aligned with intraoperative MRI.
Tensor alignment: Ruiz et al. 2000
Computational Radiology Laboratory. Slide 39
Conclusion
• Validation strategies for registration:– Comparison of transformations.– Fiducials
• Manual, automatic.
– Overlap statistics – as for segmentation.
• Validation strategies for segmentation:– Digital and physical phantoms.– Comparison to domain experts.– STAPLE.
Computational Radiology Laboratory. Slide 40
Acknowledgements
• Neil Weisenfeld.• Andrea Mewes.• Richard Robertson.• Joseph Madsen.• Karol Miller.• Michael Scott.
This study was supported by:R01 RR021885, R01 EB008015, R01 GM074068
Collaborators• William Wells.• Kelly H. Zou.• Frank Duffy.• Arne Hans.• Olivier Commowick.• Alexandra Golby.• Vicente Grau.