Medical J. Michael Fitzpatrick, Department of Electrical Engineering and Computer Science Vanderbilt University, Nashville, TN Course on Medical Image

Medical

J. Michael Fitzpatrick, Department of Electrical Engineering and Computer ScienceVanderbilt University, Nashville, TN

Course on Medical Image Registration, Nov 3-Nov 24, 2008Institute für Robotic, Leibniz UniversitätHannover, Germany

Image RegistrationImage Registration

Schedule

Nov 3: Overview of Medical Image Registration

Nov 10: Point-based, rigid registration

Nov 17: Intensity-based registration

Nov 24: Non-rigid registration

Computed Tomography (1972)

Siemens CT Scanner (Somatom AR)

3D Cross-sectional Image

“voxels” (“volume elements”)

Magnetic Resonance Imaging

GE MR Scanner (Signa 1.5T)

Positron Emission Tomography

GE PET Scanner

Physician has 3 or more views.

CT(bone)

MR(wet tissue)

PET(biologicalactivity)

Combining multiple images requires image registration

Image Registration: Definition

Determination of corresponding points in two different views

Motion relative to the scanners can be three-dimensional.

Slice orientations vary widely. transverse sagittal coronal

Views may be very different.

But all orientations and all views can be combined if we have the 3D

point mapping.

Combining Registered Images = “Image Fusion”

MR + PETCT + MRCT MR PET

Rigid Registration: Definition

Rigid Registration = Registration using a “rigid” transformation

Rigid Transformation

Rigid Non-rigid

Distances between all points remain constant.

6 degrees of freedom

Nonrigid Transformationscan be very complex!

[Thompson, 1996]

Non-rigid example

Registration Dichotomy

• “Retrospective” methods (nothing attached to patient before imaging)

Match anatomical features: e.g., surfaces Maximize similarity of intensity patterns

• “Prospective” methods (something attached to patient before imaging)

Non-invasive: Match skin markers Invasive: Match bone-implanted markers

Most Common Approaches

• Intensity-based* (not for surgical guidance)

• Surface-based (requires identified surfaces)

• Point-based (requires identified points)

• Stereotactic frames (for surgical guidance)

*Sometimes called “voxel-based”

The Most Successful Intensity-Based Method:

Mutual Information

2D Intensity Histogram (Hill94)

CT

MRCT intensity

MR

inte

nsity

Misregistration Blurs It

0 cm 2 cm 5 cm

MR

CT

MR

PET

Hill, 1994

• A measure of histogram sharpness • Most popular “intensity” method • Assumes a search method is available• Stochastic, multiresolution search common• Requires a good starting pose• May not find global optimum• Not useful for surgical guidance

Mutual Information(Viola, Collignon, 1996)

Example: Mutual Information

Studholme, Hill, Hawkes,

1996, “Automated

3D registration of MR and CT images of the head”, MIA,

1996

(Open movie with

QuickTime)

The Most Successful Surface-Based Method:

The Iterative Closest-Point Algorithm

• Minimizes a positive distance function• Assumes surfaces have been delineated• Guaranteed to converge• Requires a good starting pose• May not find global optimum• Can be used for surgical guidance

Iterative Closest-Point Method(Besl and McKay, 1992)

Start with two surfaces

Reorient one (somehow)



Pick points on moving surface

Pick points on moving surface

Remove moving surface

Points become proxy for surface

Find closest points on stationary surface

Measure the total distance

Remove stationary surface

Points become proxy for surface

Register point sets (rigid)

Register point sets (rigid)

Restore stationary surface

Find (new) closest points

Find (new) closest points



Register Points

Register Points, and so on…

Iterative Closest-Point Algorithm:

• Find closest points• Measure total distance• Register points

Stop when distance change is small.

ICP: Image-to-Image

Dawant et al.

ICP: Image to Patient

• The BrainLab VectorVision surgical guidance system uses surface-based registration.

ICP requires surface delineation, which is a problem in Image Segmentation

Example: Level Set Segmen-

tation (Dawant

et al.)

http://www.vuse.vanderbilt.edu/~dawant/levelset_examples/

The fiducial marker is used in prospective registration for image-guided surgery.

The Most Common Application of The Point-based Method:

The Fiducial Marker

Image-Guided Surgery

...and the other is the patient.

One view is an image....

Just another image registration problem.

Acustar™

Allen, Maciunas, Fitzpatrick, and

Galloway

1988-1995 (J&J Z-Kat)

are implanted into the skull.

Posts

[Maurer, et al., TMI, 1997]

Acustar™


Galloway

1988-1995 (J&J Z-Kat)


Liquid in marker

shows up

in image

Divot cap is localizable

in OR

Acustar™


Galloway

1988-1995 (J&J Z-Kat)


Marker center and cap center occupy the same position relative

to the post

Acustar™


Galloway

1988-1995 (J&J Z-Kat)


Marker center and cap center occupy the same position relative

to the post

Find corresponding

“fiducial” points

Point-based, Rigid Registration

View 2= “Space” 2

View 1= “Space” 1

Rigid transformation

Align corresponding

fiducials“targets” are also aligned

Find all corresponding

“fiducial” points

Measures of Error

View 1

Registered Views

View 2

Fiducial Localization Error (FLE)

Target Registration Error (TRE)Fiducial Registration

Error (FRE)

The Most Successful Point-based Method (by far!):

Minimization of Sum of Squares of

Fiducial Registration Errors

• Minimizes a positive distance function• Most popular point method • Assumes points have been localized• Guaranteed to converge• Does not require a good starting pose• Always finds global optimum• Can be used for surgical guidance

Minimization of Sum of FRE2

(Shönemann, Farrell, 1966)

Accuracy: State of the Art

The best accuracy is probably achieved for the head…

Retrospective Registration of Head: Image-to-Image

Median Maximum

CT-MR : 0.6 mm 3.0 mm

PET-MR: 2.5 mm 6.0 mm

TRE

Prospective Registration of Head: mean TRE ≤ 1 mm (CT)

[Hill, JCAT, 1998, Maurer, TMI, 1997]

Error Theory for Minimization of

Mean-square FRE

End of Overview

How to Do

Minimization of Sum of Squares of

Fiducial Registration Errors

Sum of Squares: Step 1

N

N

i

i

yy

xx

yyyxxx iiii ~ ; ~ :pointsCentered""

Center the points:

Centered

xy

Step 2 (Shönemann, Farrell, 1966)

)det(VU ,1 1 diag

where, D

0

),,diag(Λ

where

,Λ :SVD

~~

t

3 2 1

3 2 1

,D

UVR

IVVUU

VUH

H

t

tt

t

ii

tyx

Determine the Rotation: Centered

Centered and Rotated

Step 3 (Farrell, 1966)

R t y x

Determine the Translation:

Rx

ytx

y

Before rotation After rotation, but before translation

Error Analysis

Start with Assumptions about FLE

Independent, normal, isotropic, zero mean

Space 1 Space 2

“Effective” FLE

22221 FLEFLEFLE

1FLE

FLE

2FLESpace 1 Space 2

FRE Statistics: Sibson 1979

22

2

22

2

FLE)21(FRE

DOF. 63with

square-chi is thewhere

,FRE

:FLE To

/N

N

c

O

Approximate Solution:

Configuration doesn’t

matter!

22

2

222

2

FLE)21(FRE

DOF. 63with


,3/FLEFRE

:FLE To

/N

N

N

O

2

2 2 22 1 2 3

2 2 21 2 3

1TRE 1 / 3 FLE

d d d

N f f f

Principal axesConfiguration does

matter.

d1d2d3

[Fitzpatrick, West, Maurer, TMI, ’98]

TRE statistics, 1998Approximate Solution:

Got to here Nov 10, 2008

4mm3mm

2mm

1mm

2mm

1mm

FRE = 1mm

TRE for

FLEof

1mm

Marker Placement

[West et al., Neurosurgery, April, 2001]

A distribution would be better

<TRE2>

TRE295% level

Pro

babi

lity

den

sity

And what about direction?

TRE statistics, 2001

.directions

orthogonal along

components

t independen denote

3,2,1 where

,0, TRE i

i

N i

Approximate Solution:TRE1

TRE2TRE3

[Fitzpatrick and West., TMI, Sep 2001]

Some Remaining Problems

Isotropic Scaling

[Actually now solved: Batchelor, West, Fitzpatrick, Proc. of Med. Im. Undstnd. & Anal. ,

Jul 2002]

Anisotropic Scaling

(Iterative Solution Only)

Register M points sets simultaneouslyView 1 View 2

;

View 3 View M

The “Generalized” Procrustes Problem

(Iterative Solution Only)

Anisotropic FLE

(Iterative Solutions Only)

Other Unsolved Problems

• What is the statistical effect on TRE of dropping or adding a fiducial?

• Does anisotropy in FLE always, sometimes, or never makes TRE worse?

• How do we configure markers on a given surface so as to minimize TRE over a given region?

• Is there a correlation between FRE and TRE?It’ solved: There is no correlation!

Fitzpatrick, SPIE Medical Imaging Symposium, to be presented Feb 2009.

• Extension to perspective transformations.

• Extension to surface matching.

Other Unsolved Problems (cont.)

Rigid Registration of the Head

State of the Art

CT MR-T1 MR-T2

Finding Points = “Localization”

Acustar v. Leibinger:Leibinger Grows Up!

Retrospective Registration of Head Images: The State of the Art

Median Maximum (Acustar)

Best CT-MR : 0.6 mm 3.0 mm (0.5 mm)

Poor CT-MR: 5.4 mm 61 mm (0.5 mm)

Best PET-MR: 2.5 mm 6.0 mm (1.7 mm)

Poor PET-MR: 5.3 mm 15 mm (1.7 mm)

And how do we know?…

Retrospective Image Regstration Evaluation

Access: 150+ participants in 20 countries

Evaluation: 57 participants in 17 countries

External siteVanderbilt

1995-2007

End

Additional slides follow

Categories within error prediction

• Number of point sets: Two or more

• Scaling: Isotropic or anisotropic

• Point-wise weighting: equal or unequal

• Anisotropic weighting

• Cost function: squared error or other

• Point-wise FLE: equal or unequal

• Spatial FLE: isotropic or anisotropic...

Key: Approximate, Negligible progress

Anisotropic Scaling

R, t = rotation, translationwi

2= point weightingS = diag( sx , sy , sz )

N

iiii RSwN

22)/1( ytx

Given {xi yi wi} find R, t, S to minimize mean FRE2

.for

),,(FRE min

minimizes

that Sfor Search

i

2

R,

i

i

S

R

xx

xtt

Iterative Algorithm:

sy

sz

sx

Searchspace

Problem Statement:

Scaling: Anisotropic II


2= point weightingS = diag( sx , sy , sz )

N

iiii SRwN

22)/1( ytx


),(FRE

minimizes

that , ,for Search

:In harder thamuch

gly)(surprisin is II

2 t

t

RS,

RS

Iterative Algorithm:Problem Statement:

Spatial Weighting


2= point weightingS = diag( sx , sy , sz )A = diag( ax , ay , az )

N

iiii RSAwN

22 )()/1( ytx


[96] vTrendafilo &Chu

[91] Swane &Koschat

Iterative Algorithm:Problem Statement:

Batchelor and

Fitzpatrick [2000]

Partial Solution:

Generalized Procrustes Problem

Cost function Iterative method(only)

Add Isotropic Scaling


22

2

222

FLE3

71FRE

DOF 73 has

.3/FLEFRE

:) To

N

N

N

O( 2

FRE2 = sum of squared fiducial

registration errors

FRE: Generalized + Scaling


22

2

22

2

FLE3

71FRE

DOF. 7)-1)(3N-(M has

.13FLE

FRE :) To

N

MN

O( 2

FRE2 = sum of squared fiducial

registration errors

TRE statistics with scaling

2

To

TRE

West and Fitzpatrick [2001]

2O


TRE2 = target

registration error

Applications of TREStatistics

Surgical Paths

Radiation Isodose

Contours

Error Bounds

Probe Design

Tip = “target”

IREDs are fiducials

FLE

TRE

ix22i TREFLEFRE 2

Fiducial-Specific FRE

1x Poor fiducial alignment tends to occur where target registration is good!!

2x

Four Solution Methods

1/21. Square Root:

2. SVD: , where

3. Quaternion:

(a) Form 4x4 matrix from elements of .

(b) Find eigenvector of with largest eigenvalue.

(c) Elements of

t t

t t

R H HH

R VU H UDV

Q H

Q

R

q

1 2 3 4 are quadratic in , , , .

4. Dual-number quaternion: [Walker91].

q q q q

( All work equally well [Eggert91]! )

.~~on only depends and easy, is tyxt ii2iwHR


)()()()()(

2)()(2

)()()(

)(

where

minimize to

}1 |,,{ find

},1 1 , |{

each, points of sets For

mmi

mmmi

M

m

M

mk

N

i

ki

mii

mmm

m

SR

w

MmSR

MN,mi

NM

i

txx

xx

t

x

(We’ve already done it for M=2.)

Problem Statement: Illustration:


neglible. are changes until

. where

,,, 1 ,ˆ

minimize to,, Find

.)/1(ˆ

: thisIterate

. Start with

)()()()()(

2)(2

*)()()(

)(

)()(

mmi

mmmi

N

ii

mii

mmm

M

m

mi

mi

mi

SR

Mmw

SR

Mi

txx

xx

t

xx

xx

Iterative Algorithm: Illustration:

*Subject to S(m) normalization

Approximation Method

.

:0 , if sameoutput that Note 2

., :FLEs small Assume (1)

2

)()(

2)()(

1

21

iii

truetrue

itrue

itrue

i

iii

IR

R

gxy

t

gtzy

fzx

(due to Sibson, 1979)

Approximation Method (cont.)

orderhigher dropping ,in Expand (3)

)(TRE

)(FRE

) (

22

22

)1(

O

O

O

RIR

t

FRE Statistics

N

iii

i

RN

Ni

2

R,

2

21

1min FRE

for Statistics :Find

and

,,1for }{ :Given

ytx

z

t

Problem Statement: Approximate Solution:

22

2

222

21

22

FLE)21(FRE

freedom. of degrees

63 with ddistribute


,/FRE

:) To

/N

N

N

O( 2

TRE statistics with scaling

N

iii

s

i

RsN

Rs

Ni

2

,R,

2

2

1

1min FRE

when ,TRE

for Statistics :Find

. and ,

,,1for }{ :Given

ytx

ytx

x

t

Problem Statement:

2

To

TRE

West, Fitzpatrick,

and Batchelor [2001]

2O


What do “solved” and “unsolved” mean?

• “Solved”, working definition: Reduced to solving algebraic equations Iterative algorithm that converges to solution Approximate solution accurate to

• “Unsolved”: Not solved

)(O

Point-wise weighting: Equal or Unequal(We’ve just looked at this one.)


2= point weighting

N

iiii RwN

22)/1( ytx

Given {xi yi wi} find R, t to minimize mean FRE2

Problem Statement:

See previousslides again!

Solution:

1. Performing a Registration

xi = point in “from” set; yi = point in “to” set.t = translation vector.R = 3x3 rotation matrix (therefore RtR = I ).

Rxi + t

N

iiii RwN

22)/1( ytx

Given {xi yi wi } find R, tto minimize mean FRE2 xi yi

( usually wi=1)

a.k.a. The “Orthogonal Procrustes Problem”Problem Statement:

2. Predicting Registration Error

View 1

Registered Views

View 2

Input---•fiducial positions

•target position, r•FLE distribution

Output---statistics for

TRE

r

Output---statistics for

FRE

Isotropic Scaling

iii

iiii

w

Rw

s

R,

s

xx

yx

t

~~

~~

(2)

Find

1Set (1)

2

2


2= point weightings = isotropic scaling

N

iiii RswN

22)/1( ytx

Given {xi yi wi} find R, t, s to minimize mean FRE2

Problem Statement: Solution:

AcknowledgementsBenoit M. Dawant, PhD, EECS

Robert L. Galloway, PhD, BME

William C. Chapman, MD, Surgery

Jeannette L. Herring, PhD, EECS

Jim Stefansic, PhD, Psychology

Diane M. Muratore, MS, BME

David M. Cash, MS, BME

Steve Hartman, MS, BME

W. Andrew Bass, BME

NSF NIH

Matthew Wang, PhD, IBMJay B. West, PhD, Accuray, Inc.

Derek L. G. Hill, PhD Kings CollegeCalvin R. Maurer, Jr., PhD, Stanford U.

What could we choose to optimize?

• Mean-square “Fiducial Registration Error” (FRE2) Known as the “Orthogonal Procrustes Problem” in

statistics since 1950s.

• Robust estimators (median, M-estimators) Less sensitive to “outliers”

Color key: Major problems solved, Much less done

Documents

Medical J. Michael Fitzpatrick, Department of Electrical Engineering and Computer Science Vanderbilt University, Nashville, TN Course on Medical Image