Embed Size (px)
Citation preview
Divergence matching criteria for indexing andregistration
Alfred O. Hero
Dept. EECS, Dept Biomed. Eng., Dept. Statistics
University of Michigan - Ann Arbor
http://www.eecs.umich.edu/˜hero
Collaborators: Olivier Michel, Bing Ma, Huzefa Heemuchwala
Outline
1. Statistical framework: entropy measures, error exponents
2. α-MI indexing using single pixel gray levels
3. α-MI indexing via coincident features
4. α-entropy andα-MI estimation via MST
1
(a) ImageI1 (b) ImageI0 (c) Registration result
Figure 1: A multidate image registration example
2
Statistical Framework� I : an image
� Z = Z(I): an image feature vector over[0;1]d
� IR a reference image, featureZR
� fI ig a database ofK images, featuresZi
� f (ZR; Zi): joint feature density
For a pairZR andZi we have two hypotheses:
H0 : f (ZR; Zi) = f (ZR) f (Zi)H1 : f (ZR; Zi) 6= f (ZR) f (Zi)
3
Divergence Measures
Refs: [Csiszar:67,Basseville:SP89]
Define densities
f1 = f (ZR; Zi); f0 = f (ZR) f (Zi)
The Renyi α-divergence of fractional orderα 2 [0;1] [Renyi:61,70 ]
Dα( f1 k f0) =
1α�1
ln
Zf1
�
f1f0
�αdx
=
1α�1
ln
Zf α1 f 1�α
0 dx
4
Renyi α-Divergence: Special cases� α-Divergence vsα-Entropy
Hα( f1) =
11�α
ln
Z
f α1 dx=�Dα( f1 k f0)j f0=U([0;1]d)
� α-Divergence vs. Batthacharyya-Hellinger distance
D 12
( f1 k f0) = ln�Z p
f1 f0dx
�2
D2BH( f1k f0) =
Z �pf1�
pf0
�2dx= 2
�
1�Z p
f1 f0dx
�
� α-Divergence vs. Kullback-Liebler divergence
limα!1
Dα( f1k f0) =Z
f1 lnf1f0
dx:
5
Renyi α-divergence and Error Exponents
Observe i.i.d. sampleW = [W1; : : : ;Wn]
H0 : Wj � f0(w)
H1 : Wj � f1(w)
Bayes probability of error
Pe(n) = β(n)P(H1)+α(n)P(H0)
LDP gives Chernoff bound [Dembo&Zeitouni:98]
liminfn!∞
1n
logPe(n) =� supα2[0;1]
f(1�α)Dα( f1k f0)g :6
Registration via α-Mutual-Information
Ref: Viola&Wells:ICCV95
1. ReferenceIR and targetIT images.
2. Set of rigid transformationsfTig3. Derived feature vectors
ZR = Z(IR); Zi = Z(Ti(IT))
7
H0 : fZR;Zig independent
H1 : fZR;Zig dependent
Error exponent isα-MI (Neemuchwala&etal:ICIP01,
Pluim&etal:SPIE01)
MIα(ZR;Zi) =
1α�1
ln
Z
f α(ZR;Zi)( f (ZR) f (Zi))1�αdZRdZi :
8
Registration via α-Jensen-Difference
Ref: Ma&etal:ICIP00, He&etal:SigProc01
� Jensen’s difference btwnf0; f1:
∆Jα = Hα(ε f1+(1� ε) f0)� εHα( f1)� (1� ε)Hα( f0)� 0
� f0; f1 are two densities,ε satisfies 0� ε� 1
� Let X ;Y be i.i.d. features extracted from two images
Xm = fX1; : : : ;Xmg; Yn = fY1; : : : ;Yng
� Each realization inunorderedsampleZ = fXm; Yng has marginal
fZ(z) = ε fX(z)+(1� ε) fY(z); ε =
mn+m
9
Ultrasound Registration Example
Figure 2: Three ultrasound breast scans. From top to bottom are: case 151,
case 142 and case 162.
10
(a) ImageX1 (b) ImageX0
Figure 3: Single Pixel Coincidences (Left and right: 0o and 8o rotations)
11
Grey Level Scatterplots
50 100 150 200 250
50
100
150
200
250
50 100 150 200 250
50
100
150
200
250
50 100 150 200 250
50
100
150
200
250
50 100 150 200 250
50
100
150
200
250
Figure 4:Grey level scatterplots. 1st Col: target=reference slice. 2nd Col: target = reference+1 slice.
12
−5 −4 −3 −2 −1 0 1 2 3 4 5−0.3
−0.25
−0.2
−0.15
−0.1
−0.05
0
0.05
ANGLE OF ROTATION (deg) −−−−−−>
α−D
IVE
RG
EN
CE
(αM
I) −
−−
−−
−>
α−DIVERGENCE v/s α CURVES FOR α ∈ [0,1] FOR SINGLE PIXEL INTENSITY
α =0 α =0.1α =0.2α =0.3α =0.4α =0.5α =0.6α =0.7α =0.8α =0.9
α=0.1
α=0.9
α=0
Figure 5:α-Divergence as function of angle for ultra sound image registration of
image 142
13
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
0.005
0.01
0.015
0.02
0.025
α −−−−−−>
curv
atur
e of
α−
MI −
−−
−−
−>
CURVATURE OF: α−MI v/s α CURVE FOR SINGLE PIXEL INTENSITY
Figure 6:Resolution ofα-Divergence as function of alpha
14
Higher Level Features
Disadvantages of gray level features:
� Only depends on histogram of single pixel pairs
� Insensitive to spatial reording of pixels in each image
� Difficult to select out grey level anomalies (shadows, speckle)
� Spatial discriminants fall outside of single pixel domain
� Alternative : Spatial-feature-based indexing
15
Local Tags
(a) ImageX0 (b) ImageXi
Figure 7: Local Tag Coincidences
16
Spatial Relations Between Local Tags
N
E
SE
S
W
R
(a) ImageX0
N
E
SE
S
W
R
(b) ImageXi
Figure 8: Spatial Relation Coincidences
17
Feature: spatial tags via feature trees
Root Node
Depth 1
Depth 2
Not examined further
Figure 9:Part of feature tree data structure.
Terminal nodes (Depth 16)
Figure 10:Leaves of feature tree data structure.
18
Feature: projection-coefficient wrt ICA basis
Figure 11:Estimated ICA basis set for ultrasound breast image database
19
Simple Example
50 100 150 200 250 300 350
50
100
150
200
250
300
350
50 100 150 200 250 300 350
50
100
150
200
250
300
350
−8 −6 −4 −2 0 2 4 6 80
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Performance of higher order features v/s single pixels
Relative rotation between images
MS
T L
engt
h an
d S
hann
on M
I
MST LengthShannon MI
Figure 12:10 basis composite images with noise and MI vs Feature divergence objective functions
20
US Registration Comparisons
151 142 162 151/8 151/16 151/32
pixel 0:6=0:9 0:6=0:3 0:6=0:3
tag 0:5=3:6 0:5=3:8 0:4=1:4
spatial-tag 0:99=14:6 0:99=8:4 0:6=8:3
ICA 0:7=4:1 0:7=3:9 0:99=7:7
Table 1: Numerator =optimal values ofα and Denominator = maximum
resolution of mutualα-information for registering various images (Cases
151, 142, 162) using various features (pixel, tag, spatial-tag, ICA). 151/8,
151/16, 151/32 correspond to ICA algorithm with 8, 16 and 32 basis ele-
ments run on case 151.
21
Feature-based Indexing: Challenges� How to best select discriminating features?
– Require training database of images to learn feature set
– Apply cross-validation...
– ...bagging, boosting, or randomized selection?
� How to computeα-MI for multi-dimensional features?
– Tag space is of high cardinality:25616� 1032
– ICA projection-coefficient space is multi-dimensional continuum
– Soln 1: partition feature space and count coincidences...
– Soln 2: apply kernel density estimation and ...
– ...plug in to theα-MI or α-Jensen formula
– Soln 3: estimateα-MI or α-Jensen directly via MST.
22
Methods of Entropy/Divergence Estimation� Z = (ZR;ZT): a statistic (feature pair)
� fZig: n i.i.d. realizations fromf (Z)
Objective: Estimate
Hα( f ) =1
1�αln
Z
f α(x)dx
1. Parametric density estimation methods
2. Non-parametric density estimation methods
3. Non-parametric minimal-graph estimation methods
23
Non-parametric estimation methods
Given i.i.d. sampleX = fX1; : : : ;Xng
Density “plug-in” estimator
Hα( fn) =
11�α
ln
Z
IRdf α(x)dx
Previous work limited to Shannon entropyH( f ) =�R
f (x) ln f (x)dx
� Histogram plug-in [Gyorfi&VanDerMeulen:CSDA87]
� Kernel density plug-in [Ahmad&Lin:IT76]
� Sample-spacing plug-in [Hall:JMS86](d = 1)
� Performance degrades as densityf becomes non smooth
� Unclear how to robustifyf against outliers
� d-dimensional integration might be difficult
� ) functionf f (x) : x2 IRdg over-parameterizes entropy functional
24
Direct α-entropy estimation� MST estimator ofα-entropy [Hero&Michel:IT99]:
Hα =
11�α
lnLγ(Xn)=n�α
� Direct entropy estimator: faster convergence for nonsmooth densities
� Parameterα is varied by varying interpoint distance measure
� Optimally prunedk-MST graphs robustifyf against outliers
� Greedy multi-scale MST approximations reduce combinatorial
complexity
25
Minimal Graphs: Minimal Spanning Tree (MST)
Let Mn = M (Xn) denote the possible sets of edges in the class of acyclic
graphs spanningXn (spanning trees).
The Euclidean Power Weighted MST achieves
LMST(Xn) = minMn
∑e2Mn
kekγ:
26
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
z1
z 2
128 random samples
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
z1
z 2
MST
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
z1
z 2
128 random samples
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
z1
z 2
MST
Figure 13:
27
Convergence of MST
0 500 10000
5
10
15
20
N
MS
T le
ngth
MST length, Unif. dist. (red), Triang. dist (blue)
0 500 10000.4
0.6
0.8
N
MST normalized compensated length
Figure 14:
28
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Figure 15:Continuous quasi-additive euclidean functional satisfies “self-similarity” property on any scale.
29
Asymptotics: the BHH Theorem and entropy estimation
Theorem 1Beardwood&etal:Camb59,Steele:95,Redmond&Yukich:SPA96Let L
be a continuous quasi-additive Euclidean functional with power-exponent
γ, and letXn = fX1; : : : ;Xng be an i.i.d. sample drawn from a distribution
on [0;1]d with an absolutely continuous component having (Lebesgue)
density f(x). Then
(1)
limn!∞
Lγ(Xn)=n(d�γ)=d = βLγ;d
Z
f (x)(d�γ)=ddx; (a:s:)
Or, lettingα = (d� γ)=d
limn!∞
Lγ(Xn)=nα = βLγ;d exp((1�α)Hα( f )) ; (a:s:)
30
Asymptotics: Plug-in estimation ofHα( f )
Class of Holder continuous functions over[0;1]d
Σd(κ;c) =n
f (x) : j f (x)� pbκcx (z)j � c kx�zkκ
o
Proposition 1 (Hero&Ma:IT01) Assume that fα 2 Σd(κ;c). Then, if f α
is aminimax estimator
supf α2Σd(κ;c)
E1=p
�����Z bf α(x)dx�Z
f α(x)dx
����p� = O
�
n�κ=(2κ+d)�
31
Asymptotics: Minimal-graph estimation of Hα( f )
Proposition 2 (Hero&Ma:IT01) Let d� 2 and
α = (d� γ)=d 2 [1=2;(d�1)=d]. Assume that fα 2 Σd(κ;c) whereκ� 1
and c< ∞. Then for any continuous quasi-additive Euclidean functional
Lγ
supf α2Σd(κ;c)
E1=p
�����Lγ(X1; : : : ;Xn)nα �βLγ;d
Z
f α(x)dx
����p��O
�
n�2=(3d)�
Conclude: minimal-graph estimator converges faster for
κ <
2d3d�4
32
Observations� Minimal graph rates valid for MST,k-NN graph, TSP, Steiner Tree,
etc
� Analogous rate bound holds for progressive-resolution algorithm
LGγ (Xn) =
md
∑i=1
Lγ(Xn\Qi)
fQig is uniform partition of[0;1]d into cell volumes 1=md
� Optimal sequence of cell volumes is:
m�d = n�1=3
33
Computational Acceleration of MST
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
z0
z1
Selection of nearest neighbors for MST using disc
r=0.15
Figure 16:Acceleration of Kruskal’s MST algorithm from n2 logn to nlogn.
34
0 2000 4000 6000 8000 10000−50
0
50
100
150
200
250
300
Number of Points, N with Uniform Distribution
Comparison: Standard and Modified Kruskal Agorithms
Exe
cutio
n T
ime
in s
econ
ds
Standard Kruskal Modified Kruskal (radius=0.09)
Figure 17:Comparison of Kruskal’s MST to our nlogn MST algorithm.
35
0 0.02 0.04 0.06 0.08 0.10
10
20
30
40
50
60
70
Radius of disc
MS
T L
engt
h
Effect of disc radius on MST Length: 10000 unif. dist. points
Figure 18:Bias of nlogn MST algorithm as function of radius parameter.
36
Application of MST to US image Registration
1. Extract features from reference and transformed target images:
Xm = fXig
mi=1 and Yn = fYig
nyi=1
2. Construct MST on union ofXm andYn
Lγ(Xm[Yn)
3. MinimizeLγ over transformations producingYn.
Note: This minimizer converges to minimizer ofα-Jensen difference
Lγ(Xm[Yn)=(m+n)α ! βLγ;d exp((1�α)Hα(ε fx+(1� ε) fy)) ; (a:s:)
whereε = mm+n
37
0
200
400
0
100
200
3000
10
20
30
40
X
misaligned points
Y
Inte
nsity
(a)
0
200
400
0
100
200
3000
10
20
30
40
X
MST demonstration
Y
Inte
nsity
(b)
Figure 19: MST demonstration for misaligned images
38
0
200
400
0
100
200
3000
20
40
60
X
Aligned points
Y
Inte
nsity
(a)
0
200
400
0
100
200
3000
20
40
60
X
MST demonstration
Y
inte
nsity
(b)
Figure 20: MST demonstration for aligned images
39
Illustration for Case 142 and Single Pixels
−8 −6 −4 −2 0 2 4 6 80
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Relative rotation between images (degrees)
Nor
mal
ized
MS
T L
engt
h, M
I
MST Length, alphaMI profiles. Single Pixel domain, vol. 142
MI (alpha=0.5)MST Length
Figure 21:Single-pixel objective function profiles for MST estimator ofα-Jensen difference vs histogram plug-in estimator (α = 1=2).
40
Illustration for Case 142 and 8-D ICA Features
−8 −6 −4 −2 0 2 4 6 80
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Relative rotation between images (degrees)
MS
T L
en.,
alph
aJen
sen(
4 bi
ns/d
imen
sion
)
MST & alphaJensen profile for 8D ICA feature vector, vol. 142
Jensen (alpha=0.5)MST Length
Figure 22: 8-basis ICA objective function profiles for MST estimator ofα-Jensen difference vs histogram plug-in estimator ofα-MI
(α = 1=2).
41
Illustration for Case 142 and 64-D ICA Features
−8 −6 −4 −2 0 2 4 6 80
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
MS
T L
engt
h
Relative rotation between images (degrees)
Norm. MST Length profile for 64D ICA feature vector, vol. 142
Figure 23:64-basis ICA objective function profiles for MST estimator ofα-Jensen difference.
42
Extension of MST to divergence estimation
1. Let i.i.d.fZig
ni=1 have marginal densityf1 on [0;1]d
2. Let f0 dominate densityf1
3. Define measure transformationM on [0;1]d which takesf0 to uniformdensity
ThenSidef
= M (Zi) hasα-entropy:
(1�α)Hα(Si) = ln
Zf αS (s)ds= ln
Z( f1(z)= f0(z))
α f0(z)dz
Conclude, forSn = fS1; : : : ;Sng:
Dα( f1k f0) =
11�α
�
lnLγ(Sn)=nα� lnβL;γ
�: (2)
is a consistent estimator ofα-divergence
43
Example
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1Original data
z1
z 2
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
exact inverse transform
z1
z 2
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1tranfd data
z1
z 2
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1tranfd data, 2D cdf estd
z1
z 2
Figure 24:
44
MST robustification against outliers: Pruned MST
Fix k, 1� k� n.
Let Mn;k = M (xi1; : : : ;xik) be a minimal graph connectingk distinct
verticesxi1; : : : ;xik.
Thek-MST T�n;k = T�(xi�1
; : : : ;xi�k
) is minimum of allk-point MST’s
L�n;k = L�(Xn;k) = mini1;:::;ik
minMn;k
∑e2Mn;k
kekγ
45
0 0.5 1
0
0.2
0.4
0.6
0.8
1
z2
k−MST (k=99): 1 outlier rejection
0 0.5 1
0
0.2
0.4
0.6
0.8
1
z2
(k=98): 2 outlier rejection
0 0.5 1
0
0.2
0.4
0.6
0.8
1
z1
z2
k−MST (k=62): 38 outlier rejection
0 0.5 1
0
0.2
0.4
0.6
0.8
1
(k=25): 75 outlier rejection
z1z2
Figure 25:k-MST for 2D annulus density with and without the addition of
uniform “outliers”.
46
Extension of BHH to Pruned Graphs
Fix α 2 [0;1] and letk= bαnc. Then asn! ∞ (Hero&Michel:IT99)
L(X �
n;k)=(bαnc)ν ! βLγ;d minA:P(A)�α
Z
f ν(xjx2 A)dx (a:s:)
or, alternatively, with
Hν( f jx2 A) =1
1�νln
Z
f ν(xjx2 A)dx
L(X �
n;k)=(bαnc)ν ! βL;γ exp
�(1�ν) min
A:P(A)�αHν( f jx2 A)
�(a:s:)
47
Water Pouring Construction
0 10 20 30 40 500
0.1
Density of X
x
f(x)
0 10 20 30 40 500
0.1
x
max
(f(x
)−f(
x)
Water filled density
0 10 20 30 40 500
0.1
0.2Asymptotic density of X seen by k−MST
x
f(x|
A)
Figure 26:Water pouring construction of f(xjAo). Arrows indicate the high water marks indicating high probability regions which are
not truncated in the influence function.
48
k-MST Influence Function for Gaussian Density
−20
0
20
−20
0
20−100
0
100
200
300
MST for planar Gaussian
IC(x
,F,L
)
−20
0
20
−20
0
20−40
−20
0
20
40
60
k−MST for planar Gaussian
IC(x
,F,L
)
Figure 27:MST and k-MST influence curves for Gaussian density on the plane.
49
k-MST Stopping Rule
0 20 40 60 80 1000
20
40
60
80
100
k
k−M
ST
leng
thk−MST length as function of k
0 20 40 60 80 1000
5
10
15
20
25
30
k
crite
rion(
k)
selection criterion
Figure 28:Left: k-MST curve for 2D annulus density with addition of uniform “outliers” has a knee in the vicinity of n� k = 35. This
knee can be detected using residual analysis from a linear regression line fitted to the left-most part of the curve. Right: error residual of linear
regression line.
50
Conclusions
1. α-divergence for indexing can be justified via decision theory
2. Applicable to feature-based image registration
3. Non-parametric estimation is possible without density estimation via
MST
4. MST outperforms plug-in estimation for non-smooth densities
5. Robustified MST can be defined via optimal pruning of MST: k-MST
51
Divergence vs. Jensen: Asymptotic Comparison
For ε 2 [0;1] andg a p.d.f. define
fε = ε f1+(1� ε) f0; Eg[Z] =Z
Z(x)g(x)dx; f α12
=
f α12R
f α12dx
Then
∆Jα =
αε(1� ε)
2
24Ef α12
0@" f1� f0f 1
2
#2
1A+
α1�α
Ef α12
"
f1� f0f 1
2
#!2
35+O(∆)
Dα( f1k f0) =
α4
Z
f 12
"
f1� f0f 1
2
#2
dx+O(∆)
52
