Multimodal Registration of Medical Data

Prof. Leo Joskowicz

School of Computer Science and Engineering

The Hebrew University of Jerusalem

Intensity-based rigid registration (1)

• Use intensity information to define the measure of similarity between two data sets

• Rationale: the closer the data sets are, the more similar their intensity values are.

• No segmentation is necessary! The entire data set is used. Slow, especially for 3D data sets.

• The parametric space of transformations is searched incrementally from an initial configuration. The search space is six-dimensional (3 rotations and 3 translations)

Intensity-based registration (2)• Similarity measures

– cross correlation– histogram correlation– mutual information– intensity values

• Uses: brain CT/MRI, Xray/CT• Example: fluoroscopic Xray to CT

Xray/CT registration

• Problem definition: given– preoperative CT data set of rigid structure– intraoperative Xray images from a calibrated

camera at relatively known spatial configurations

• Find a rigid transformation that matches the CT data set to the intraoperative Xrays so that if Xrays of the CT were taken from the transformed camera positions, the resulting Xray images would be identical to the intraoperative ones.

Xray/CT registration setup

2D/3D registration problem!

Ref C-arm

Ref patientCt slices

Ref ct volume

DRR

Fluoroscopic image

Xray to CT registration algorithm

Input: preop CT, intraop Xray Ifluoro , intrinsic Xray camera parameters, initial guess p0 for camera pose

1. generate simulated Xray IDRR (called digitally reconstructed radiograph, or DRR) at camera pose pi

2. Compute dissimilarity between IDRR and Ifluoro by comparing their intensities

3. Compute a new camera pose pi+1 = pi + d that best reduces the dissimilarity between IDRR ,and Ifluoro)

repeat until no progress can be made

Digitally reconstructed radiographs

Generating DRRs

For each pixel in the DRR plane, construct the ray emanating from the camera focal point. Sum up the intensities of the CT voxel values according to the Xray attenuation formula to obtain the gray level value of the DRR pixel.

DRR

CT

Camera

Generated DRRs

Real X-ray vs DRR

Similarity measure Pairwise comparison of normalized pixel

intensity values:• IDRR(i,j) and Ifluoro (i,j) are the pixel values

• IDRR and Ifluoro are the average image values

• T is the region of interest

Examples of initial poses registrations (DRRs only)

Actual use: radiation therapy with the Cyberknife (radiation therapy)

Frameless radiation therapyStereotactic setup

Track head with Xraysbefore each dose application

Matching skull X-ray and DRR

Match only regions

Intensity-based registration

• Advantages – no segmentation, automatic– selective regions – potentially accurate

• Disadvantages– large seach space, many local minima– slow

Deformable registration: scope

• Necessary for soft tissue organs and for cross-patient comparisons– brain images before and during surgery– anatomical structures at different times or from

patients: tumor growth, heart beating, compare– matching to atlases

• Much more difficult than rigid registration!– problem is ill-posed; solution is not unique– error measurements and comparisons are difficult– local vs. global deformations?

Deformable registration: properties• Mapping transformation can be

– global, e.g., a bi- or tri-variate polynomial– local, e.g.a fine grid with displacement vectors

• Define an energy function that should be minimized to make the data sets match.

• Usually comes after rigid registration to get an approximate position estimate.

• Both geometry based and intensity-based techniques exist.

rigid affine triliear quadratic

quadratictransform

Mathematics of deformationsGlobal transformations

Local spline deformation

Square shift

Global deformation transformations

Scale

Local grid-based deformable registration

image 1 image 2image 1+2 deformation map

Example: MRI slice matching image 1 image 2 after registration

difference image without deformation

difference image with deformation

Brain tumor matching - 2D map

Brain tumor matching - 3D map

source target match

Initial configuration

After rigid registration

After deformableregistration with local splines

Example: spine matching

Deformable registration techniques

• Too many to list here!– Optical flow model– Physics-based: elastic and fluid models

• Use an elastic or deformable model

• Validation is difficult

Commercial products• Medical image processing software packages

include some registration capabilities (manual or semi-automatic feature selection)

• Contact-based rigid registration of CT and optical tracker in orthopaedics and neurosurgery (half a dozen companies)

• Intraoperative Open MR to tracker rigid registration

The future: research directions• In many areas, the problem is far from solved:

similar to image segmentation!• Much clinical validation is needed. More

coverage of other anatomy (60% focus on brains!)

• Interleave segmentation and registration

• Difficult data sets: 2D and 3D ultrasound images, video sequences, portal images

• Model-based techniques are the most likely to be sufficiently robust for clinical use

• Integration requirements are very important.

Bibliography (1)• Two chapters on registration in Computer-Integrated

Surgery, Taylor et al, MIT Press, 1995. • Medical Image Registration, Hajnal et al, CRC Press 2001• “A survey of medical image registration”, Maintz and

Viergever, Medical Image Analysis Journal, 2(1), Oxford University Press 1998 (over 150 references!)

• “A method for registration of 3D shapes”, Besl and McKay, IEEE Trans. on Pattern Analysis, 14(2), 1992.

• Special issue on Biomedical Image Registration, Image and Vision Computing, Vol 19(1-2), 2001.

• “Deformable models in medical image analysis: a survey”, McInerney and Terzopolous, Medical Image Analysis 1(2), 1996.

Bibliography (2)• “Retrospective registration of tomographic brain

images”, J. Mainz, PhD Thesis, Utrecht U., 1996 www.cs.ruu.nl/people/twan/personal/list.html

• “Localy affine registration of free-form surfaces”, J. Feldmar and N. Ayache, Proc. IEEE CVPR , 1994.

• “Matching 3D anatomical surfaces with non-rigid deformations using octree splines”, R. Szeliski and S. Lavallee, Int. Journal of Computer Vision 18(2), 1996.

• “Fast intensity-based non-rigid matching”, P. Thirion, Proc. Conf. Medical Robotics and CAS, 1995.

• “Multimodal volume registration by maximization of mutual information, W.Wells, P.Viola, R.Kikinis. (idem)