Medical Image RegistrationMedical Image Registration

Yujun GuoYujun Guo

Dept.of CSDept.of CS

Kent State UniversityKent State University

OutlineOutline

Why registrationWhy registration

Registration basicsRegistration basics

Rigid registrationRigid registration

Non-rigid registrationNon-rigid registration

ApplicationsApplications

Modalities in Medical ImageModalities in Medical Image

Computed Tomography (CT), Magnetic Resonance (MR) imaging, Ultrasound, and X-ray give anatomic information.

Positron Emission Tomography (PET) and Single Photon Emission CT (SPECT) give functional information.

RegistrationRegistration

• Monomodality:Monomodality: A series of same modality A series of same modality images (CT/CT, MR/MR, images (CT/CT, MR/MR,

Mammogram pairs,…). Mammogram pairs,…). Images may be acquired weeks or months apart; Images may be acquired weeks or months apart;

taken from different viewpoints.taken from different viewpoints. Aligning images in order to detect subtle changes in Aligning images in order to detect subtle changes in

intensity or shapeintensity or shape

• Multimodality:Multimodality: Complementary anatomic and functional information Complementary anatomic and functional information

from multiple modalities can be obtained for the from multiple modalities can be obtained for the precise diagnosis and treatment.precise diagnosis and treatment.

Examples:PET and SPECT (low resolution, functional Examples:PET and SPECT (low resolution, functional information) need MR or CT (high resolution, information) need MR or CT (high resolution, anatomical information) to get structure anatomical information) to get structure information.information.

Registration Problem DefinitionRegistration Problem Definition

p = (825,856)

q = (912,632)

q = T(p;a)

Pixel location in first image Homologous pixel location in second image

Pixel location mapping function

Example Mapping FunctionExample Mapping Function

p = (825,856)

q = (912,632)

Pixel scaling and translation

Image RegistrationImage Registration

Define a transform T that will map Define a transform T that will map one image onto another image of the one image onto another image of the same object such that some image same object such that some image quality criterion is maximized.quality criterion is maximized.

A mapping between two images both A mapping between two images both spatially and with respect to intensityspatially and with respect to intensity

II22 = g (T(I = g (T(I11))))

Registration SchemeRegistration Scheme

ComponentsComponents

Feature SpaceFeature Space

Search Space or transformationSearch Space or transformation

Similarity MetricSimilarity Metric

Search StrategySearch Strategy

Feature SpaceFeature SpaceGeometric landmarks:Geometric landmarks:Points Points Edges Edges Contours Contours Surfaces, etc.Surfaces, etc.Intensities:Intensities:Raw pixel valuesRaw pixel values

23 35

24 56

Feature-based Feature-based Intensity-basedIntensity-based

Image transformationsTransformation

Inputimage

Output

image

w

y

x

w

y

x

876

543

210

'

'

'

mmm

mmm

mmm

100

543

210

mmm

mmm

affineM

Affine transformation

100

cossin

sincos

y

x

rigid t

t

M

Rigid transformationOriginal

shape

Rigid

Non-rigid

Similarity MetricSimilarity Metric

Absolute differenceAbsolute difference

SSD (Sum of Squared Difference)SSD (Sum of Squared Difference)

Correlation CoefficientCorrelation Coefficient

Mutual Information / Normalized Mutual Information / Normalized Mutual InformationMutual Information

Search StrategySearch Strategy

Powell’s direction set methodPowell’s direction set method

Downhill simplex methodDownhill simplex method

Dynamic programmingDynamic programming

Relaxation matchingRelaxation matching

Hierarchical techniquesHierarchical techniques

Multi-modality Brain image Multi-modality Brain image registrationregistration

Intensity-basedIntensity-based3D/3D Rigid transformation, DOF=6 3D/3D Rigid transformation, DOF=6 (3 translations, 3 rotations)(3 translations, 3 rotations)Maximization of Normalized Mutual Maximization of Normalized Mutual InformationInformationSimplex DownhillSimplex DownhillMulti-resolutionMulti-resolutionDataset: Vanderbilt UniversityDataset: Vanderbilt University

http://www.vuse.vanderbilt.edu/~image/registration/http://www.vuse.vanderbilt.edu/~image/registration/results.htmlresults.html

Mutual Information as Similarity Mutual Information as Similarity MeasureMeasure

Mutual informationMutual information is applied to measure the is applied to measure the statistic dependence between the image intensities of statistic dependence between the image intensities of corresponding voxels in both images, which is assumed to corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned.be maximal if the images are geometrically aligned.

a b BA

ABAB bPaP

baPbaPBAI

)()(

),(log),(),(

)|()(

)|()(

),()()(

ABHBH

BAHAH

BAHBHAH

Normalized Mutual InformationNormalized Mutual Information

Extension of Mutual InformationExtension of Mutual Information

Maes et. al.:Maes et. al.:

Studholme et. Al.:Studholme et. Al.:

Compensate for the sensitivity of MI to Compensate for the sensitivity of MI to changes in image overlapchanges in image overlap

)()(

)(2),(

),(),(),(

BHAH

AMIBANMI

BAMIBAHBANMI

),(

)()(),(

BAH

BHAHBANMI

Geometry TransformationGeometry Transformation Image Coordinate transform:

The features (dimension, voxel size, slice spacing, gantry tilt, orientation) of images, which are acquired from different modalities, are not the same.

From voxel units (column, row, slice spacing) to millimeter units with its origin in the center of the image volume.

11000

0

0

0

1

'

'

'

),,(222120

121110

020100

z

y

x

aaa

aaa

aaa

z

y

x

zyxT

Target Image & Template ImageTarget Image & Template Image

Target Image Grid

j

i

y

x

Target ImagePhysical Coordinates

y’

x’

Template ImagePhysical Coordinates

Template Image Grid

j

i

Space Transform

Images from the same patientImages from the same patient

Images provided as part of the project: “Retrospective Image Registration Evaluation”, NIH, Project No. 8R01EB002124-03, Principal Investigator, J. Michael Fitzpatrick, Vanderbilt University, Nashville,

TN.

256 x 256 pixels

MRI-T2

128 x 128 pixels

PET

Target Image ?

Template Image ?

InterpolationInterpolation

Nearest NeighborNearest Neighbor

Tri-linear InterpolationTri-linear InterpolationPartial-Volume Interpolation Partial-Volume Interpolation

Higher order partial-volume Higher order partial-volume interpolationinterpolation

Evaluating similarity measure for Evaluating similarity measure for each transformationeach transformation

y

Template Image

Transform

x

y

Target Image

x

OptimizationOptimization

Powell’s Direction Set methodPowell’s Direction Set method

Downhill Simplex methodDownhill Simplex method

Multi-resolutionMulti-resolution

Why Multi-resolutionWhy Multi-resolution Methods for detecting optimality can not guarantee that Methods for detecting optimality can not guarantee that a global optimal value will be found.a global optimal value will be found.

Time to evaluate the registration criterion is proportional Time to evaluate the registration criterion is proportional to the number of voxels.to the number of voxels.

The result at coarser level is used as the starting The result at coarser level is used as the starting point for the finer level.point for the finer level.

Currently multi-resolution approaches:Currently multi-resolution approaches:Sub-sampling Sub-sampling

AveragingAveraging

WaveletWavelet

Registration Result (I)Registration Result (I)

A typical superposition of CT-MR images.

Left : before registration Right: after registration.

Rigid transformation (II)Rigid transformation (II)

A typical superposition of MR-PET images.

Left : before registration Right: after registration.

MammographyMammographyBreast cancer is the second leading cause Breast cancer is the second leading cause of death among women in USA.of death among women in USA.Detected in its early stage, breast cancer Detected in its early stage, breast cancer is most treatable.is most treatable.MammographyMammography is the main tool for is the main tool for detection and diagnosis of breast detection and diagnosis of breast malignances.malignances.It reduces breast cancer mortality by 25% It reduces breast cancer mortality by 25% to 30% for women in the 50 to 70 age to 30% for women in the 50 to 70 age group group

Mammogram RegistrationMammogram Registration

Temporal/bilateral mammograms Temporal/bilateral mammograms varyvary– Breast compressionBreast compression– Breast positionBreast position– Imaging TechniqueImaging Technique– Change in BreastChange in Breast

Mammogram registration Mammogram registration techniquestechniques

Whole breast area vs. regionalWhole breast area vs. regional

Nipple locationNipple location

Control-point locationControl-point location

Rigid & non-rigid registrationRigid & non-rigid registration

Non-rigid Mammogram RegistrationNon-rigid Mammogram Registration

Intensity-basedIntensity-based

Elastic transformationElastic transformation

Multi-resolutionMulti-resolution

Demons algorithm (Thirion, 1996)Demons algorithm (Thirion, 1996)

DemonsDemons

Scene (Target)

Model (Template)

Transform

Demons (Cont.)Demons (Cont.)

Scene

Model

Transform

Forces

Demons (Cont.)Demons (Cont.)

Scene

Gradient

Intensity

Space

Desired Displacement

CurrentEstimation

DemonsDemons From Optical Flow From Optical Flow

Scene: f, Model: gScene: f, Model: g

Assumption: The intensity of a Assumption: The intensity of a moving object is constant with time moving object is constant with time

(1)

(2)

Description of the ApproachDescription of the Approach1.1. Select demon points.Select demon points.2.2. Compute the force Compute the force uu on the model on the model

at each of the selected demonsat each of the selected demons3.3. Determine a global transformation Determine a global transformation

based on the computed based on the computed uu and apply and apply it to the modelit to the model

4.4. If the model images is now If the model images is now registered to the scene image, stop. registered to the scene image, stop. Else, go to Step 2.Else, go to Step 2.

Registration ComponentsRegistration Components

Image IntensitiesImage Intensities

Non-rigid transformation, one Non-rigid transformation, one displacement vector for each pixeldisplacement vector for each pixel

Bilinear interpolationBilinear interpolation

Absolute difference as similarity Absolute difference as similarity metricmetric

Multi-resolutionMulti-resolution

Dataset: MIAS,DDSMDataset: MIAS,DDSM

Demons Results (I) Synthetic Images

Level=2

Level=3

Level=4

Level=5

Demons Result (II) MIASDemons Result (II) MIAS

Before registration

After rigid registration

Original images

After non-rigid registration

Ongoing registration topicsOngoing registration topics

Trade-off of computation and Trade-off of computation and accuracyaccuracy

Evaluation of registration resultsEvaluation of registration results

Visualization of registrationVisualization of registration

Applications: Change DetectionApplications: Change Detection

Images taken at different timesImages taken at different times

Following registration, the Following registration, the differences between the images may differences between the images may be indicative of changebe indicative of change

Deciding if the change is really there Deciding if the change is really there may be quite difficult may be quite difficult

Other ApplicationsOther ApplicationsMulti-subject registration to develop Multi-subject registration to develop organ variation atlases.organ variation atlases.– Used as the basis for detecting Used as the basis for detecting

abnormal variationsabnormal variations

Object recognition - alignment of Object recognition - alignment of object model instance and image of object model instance and image of unknown object (segmentation)unknown object (segmentation)

ReferencesReferencesMaes F,Collignon A, et al. “Multimodality image Maes F,Collignon A, et al. “Multimodality image registration by maximization of mutual registration by maximization of mutual information.” information.” IEEE Trans. Med. ImagingIEEE Trans. Med. Imaging. 1997, . 1997, V16,pp187-198V16,pp187-198

L.G.Brown, “A survey of image registration L.G.Brown, “A survey of image registration techniques,” techniques,” ACM Computing SurveysACM Computing Surveys, vol. 24, , vol. 24, no. 4, pp. 325–376, 1992.no. 4, pp. 325–376, 1992.

Jean-Philippe Thirion, “Non-Rigid Matching Using Jean-Philippe Thirion, “Non-Rigid Matching Using Demons,” IEEE Conference on Computer Vision Demons,” IEEE Conference on Computer Vision and Pattern Recognition,1996and Pattern Recognition,1996