Multigrid methods for variational approaches in image ... · Multigrid methods for variational approaches in image processing H. Kostler Friedrich-Alexander Universit¨at Erlangen-Nurn¨

Multigrid methods for variational approaches in imageprocessing

H. KostlerFriedrich-Alexander Universitat Erlangen-Nurnberg,Germany

January 15, 2007

H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 1 / 31

Contents

1 Multigrid framework for variational approaches

2 A curvature based optical flow regularizerOptical flow modelExperimental results

3 A variational approach for video codingVideo CompressionVideo DecompressionExperimental results

4 Medical image segmentationModelExperimental results

5 Future work


Multigrid framework for variational approaches

Goals and applications

Many efficient implementations for different variational image processingproblems exist, we try to implement a software package to support thedevelopment and efficient implementation of new variational approaches.Applications:

Optical flow

Non-rigid image registration

Tomographic image reconstruction

Image Inpainting

Image segmentation

Video compression

Motion Blur

Image denoising



Problem formulation

General problem

We try to minimize the energy functional

E [u] := D[I,u]︸︷︷︸data term

+ α S[u]︸︷︷︸regularizer

(1)

with weighting parameter α ∈ R+. We regularize the problem to get aunique solution.



Multigrid framework I

Parallel C++ Code

Based on a robust and efficient multigrid solver

Figure: Schematic application flow.



Multigrid framework II: Features

Different smoothers

Matrix dependent transfer operators

Cell-centered, node-based and staggered grids

Parallelization in 2D and 3D

Iterant recombination

Galerkin or standard coarsening

Hardware optimized solver components for special cases


A curvature based optical flow regularizer Optical flow model

Optical flow problem

Figure: The optical flow at the pixel (x,y) is the 2D-velocity vector (anapproximation to the real motion) (u, v) = ( dx

dt , dydt ).



Variational approach I

Ek(u) :=

∫Ω(Ixu + Iyv + It)

2 + αSk(u)dx (2)

Data term is brightness constancy assumption. Regularizers are diffusion(Horn and Schunck)

S1(u) = ‖∇u‖2 + ‖∇v‖2

or curvatureS2(u) = (∆u)2 + (∆v)2

that is a special case of the div-curl based regularizer [GP96]

S2′(u) = α1‖∇divu‖2 + α2‖∇curlu‖2 ,

where α1 = α2 = 1.



Variational approach II

The resulting Euler-Lagrange equations constituting a necessary conditionfor a minimum of Ek(u), k ∈ 1, 2 are

−α∆ku + Ix(Ixu + Iyv + It) = 0 (3a)

−α∆kv + Iy (Ixu + Iyv + It) = 0 (3b)

with natural Neumann boundary conditions.



Curvature based regularizer: Why?

Many different regularizers exist for optical flow, why do we introduceanother one?

Could be useful for moving fluids, can be extended to div-curlregularizer

Necessary for proper mathematical treatment of landmarks in opticalflow models (e.g. for Motion Blur)



Curvature based regularizer: Implementation

For the curvature based regularizer we introduce a variable w = ∆u andafter discretization of the Euler-Lagrange equations one has to solve thelinear system

Lh(x)

uh(x)vh(x)w1

h (x)w2

h (x)

=

00

−Ix(x, t)It(x, t)−Iy (x, t)It(x, t)

with x ∈ Ωh ,

Lh(x) =

−(1− β)∆h − β 0 1 0

0 −(1− β)∆h − β 0 1I 2x (x, t) Ix(x, t)Iy (x, t) −α∆h 0

Ix(x, t)Iy (x, t) I 2y (x, t) 0 −α∆h

.



Curvature based regularizer: Extensions

Affine transformations are in the kernel of the curvature regularizer

Combine diffusion and curvature regularizer using β ∈ [0, . . . , 1]

S3(u) = βS1(u) + (1− β)S2(u) .

Isotropic curvature regularizer

S4(u) = g(|∇I |2)((∆u)2 + (∆v)2

)with

g(s2) =1

2√

s2 + ε2.


A curvature based optical flow regularizer Experimental results

Moving Rectangle Results

Figure: Rectangle moving to the right.

Resulting velocity field of diffusion regularizer (Horn and Schunck)Resulting velocity field of curvature regularizer



Yosemite Results I

Figure: Frame 8 of Yosemite sequence with clouds (where the brightnessconstancy assumption is not fulfilled) and ground truth motion field (frames 8–9).



Yosemite Results II

50

55

60

65

70

75

140 145 150 155 160 165 170 175 180 185

50

55

60

65

70

75

140 145 150 155 160 165 170 175 180 185

Figure: Optical flow for Frames 8 and 9 from Yosemite sequence using diffusion(left) and combined (β = 0.4) (right) regularizer.



Yosemite Results III

50

55

60

65

70

75

140 145 150 155 160 165 170 175 180 185

50

55

60

65

70

75

140 145 150 155 160 165 170 175 180 185

Figure: Optical flow for Frames 8 and 9 from Yosemite sequence usinghomogeneous (left) and isotropic (right) curvature regularizer.



Yosemite Results IV: Average Angular error

8.6

8.8

9

9.2

9.4

9.6

9.8

10

10.2

10.4

10.6

0 0.2 0.4 0.6 0.8 1

AA

E

beta

AAE for alpha=500AAE for alpha=1500AAE for alpha=5000

Figure: AAE plot of the calculated optical flow between pictures 8 and 9 from theYosemite sequence for α = 500, α = 1500 and α = 5000.


A variational approach for video coding Video Compression

Compression scheme

Idea for video compression [GWW+05]

Some points are selected from each frame, such that they are sufficient fora good reconstruction of the original video sequence. We call these pointslandmarks. The rest of the points is dropped.

B-Tree Triangular Coding: fast recursive subdivision scheme todetermine landmarks

Huffman coding to store values

We do NOT (so far):

Vector quantization of colorsUse temporal information


A variational approach for video coding Video Decompression

Decompression scheme I

Solve(1− c(x))Lu − c(x)(u − f ) = δtu (4)

c(x) =

1 x ∈ Ω1

0 else(5)

f (x) =

v(x) x ∈ Ω1

0 else(6)

From these equations it follows that for every point x ∈ Ω1: u = v = f .At every other point x ∈ Ω \ Ω1 the differential equation Lu = 0 has to besolved.


A variational approach for video coding Video Decompression

Decompression scheme II

For the regularizer L we use homogeneous diffusion (HD)

Lu := ∆u , (7)

nonlinear isotropic diffusion (NID)

Lu = div(g(|∇u|2)∇u) (8)

and edge enhancing diffusion (EED)

Lu = div(g(∇uσ∇Tuσ

)∇u). (9)


A variational approach for video coding Experimental results

Compression Results I

Figure: One Frame from the original video sequence with marked parts forzooming showing water dropped in a glass.



Compression Results II

Figure: Improvement achieved by BTTC based selection. Original (UL), BTTC8% (UR), random 8% (LR), random 20 % (LL).



Compression Results III

K 1

.1

I 0.3

5K

1.1

I 0

.50

K 1

.1

I 0.9

3K

1.7

I 0

.93

K 2

.1

I 1.2

3K

2.7

I 1

.49

K 3

.2

I 2.0

0m

peg

mjp

eg

0

500

1000

1500

File

size

in K

b

Figure: Filesizes of Mpeg, Mjpeg and Pdevc - K: Keyframe Bit per pixel, I: Bitper pixel in inner frames.



Decompression Results

Figure: Comparison between homogeneous, nonlinear isotropic and nonlinearanisotropic (edge enhancing) diffusion model.



Decompression Runtimes

Regularizer Gauss-Seidel Multigrid

HD 1.2 3.7NID 0.05 0.55EED 0.03 0.43

Table: Comparison of decompression times (in fps) on a Pentium M, 2 GHzLaptop for different regularizers using Gauss-Seidel or multigrid iteration for aresolution of 320× 240 pixels.

An optimistic estimate for the maximum speed of the HD regularizer is 50fps measured for a similar setting on a Core2 Duo, 2.4 GHz (Conroe) usinga hardware optimized multigrid solver.



Video quality

(a) MPEG 0.245 bpp (b) PDEVC 2.2 bpp

Figure: Image quality for Mpeg and Pdevc (HD).


Medical image segmentation Model

Stochastic model I

Idea for image segmentation based on random walks [Gra06]

First landmarks are set manually for presegmentation. Then an isotropicdiffusion is used to determine the segmented regions. The model isstochastic, the solution we compute gives the probability for each point tobelong to a certain region.


Medical image segmentation Model

Stochastic model II

The energy functional (Dirichlet integral)

ED [u] =1

2

∫Ω|∇u|2dx (10)

has the combinatorical formulation

ED [u] =1

2xTLx =

∑eij∈E

wij(xi − xj)2 (11)

with wij = e−β(I (xi )−I (xj )). L is a combinatorical Laplacian matrix, eij areedges of a graph that corresponds to the discretization grid. Note thesimilarity to a discretized isotropic linear diffusion model.


Medical image segmentation Experimental results

Results on medical images

Figure: Half-automatic segmentation result using 5 V(2,2)-cycles (right) for anabdominal scan with marked region (left).


Future work

Future work

Further evaluate curvature based optical flow regularizer

Improve compression scheme for video coding, evaluate practical use

Hardware optimized multigrid implementation to achieve real timevideo decompression and image segmentation

Next project: Medical image denoising given several noisy images ofthe same object


Future work

Related Literature

S. Gupta and J. Prince.Stochastic models for div-curl optical flow methods.IEEE Signal Processing Letters, 3(2):32–34, 1996.

L. Grady.Random Walks for Image Segmentation.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINEINTELLIGENCE, 28(11):1, 2006.

I. Galic, J. Weickert, M. Welk, A. Bruhn, A. Belyaev, and H.P. Seidel.Towards PDE-based image compression.Proceedings of Variational, geometric, and level set methods incomputer vision, Lecture notes in computer science, pages 37–48,2005.


Documents

Multigrid methods for variational approaches in image ... · Multigrid methods for variational approaches in image processing H. Kostler Friedrich-Alexander Universit¨at Erlangen-Nurn¨