박 사 학 위 논 문
Doctoral Thesis
편미분방정식 기반의 영상 분할 방법과 복원 방법
PDE-based image processing for segmentation
and image restoration
한 주 영 (韓 周 寧 Hahn, Jooyoung)
수리과학과
Department of Mathematical Sciences
한 국 과 학 기 술 원
Korea Advanced Institute of Science and Technology
2008
편미분방정식 기반의 영상 분할 방법과
복원 방법
PDE-based image processing for segmentation
and image restoration
PDE-based image processing for segmentation
and image restoration
Advisor : Professor Lee, Chang-Ock
by
Hahn, Jooyoung
Department of Mathematical Sciences
Korea Advanced Institute of Science and Technology
A thesis submitted to the faculty of the Korea Advanced
Institute of Science and Technology in partial fullfillment of
the requirements for the degree of Doctor of Philosophy in the
Department of Mathematical Sciences
Daejeon, Korea
2007. 11. 29.
Approved by
Professor Lee, Chang-Ock
Major Advisor
편미분방정식 기반의 영상 분할 방법과
복원 방법
한 주 영
위 논문은 한국과학기술원 박사학위논문으로 학위논문심사
위원회에서 심사 통과하였음.
2007년 11월 29일
심사위원장 이 창 옥 (인)
심사위원 김 홍 오 (인)
심사위원 권 길 헌 (인)
심사위원 박 현 욱 (인)
심사위원 서 진 근 (인)
DMAS
20025321
한 주 영. Hahn, Jooyoung. PDE-based image processing for segmentation
and image restoration. 편미분방정식 기반의 영상 분할 방법과 복원 방
법. Department of Mathematical Sciences. 2008. 111p. Advisor Prof. Lee,
Chang-Ock. Text in English.
Abstract
We propose noble ideas and formulations based on nonlinear partial differential
equations (PDEs) in image segmentation and image restoration. Two algorithms
which have different perspectives on taking initial contours are proposed in image
segmentation. The first is to place initial contours arbitrarily for the purpose of cap-
turing multiple junctions and holes of objects in an image. The second is to place
those close to boundaries of objects for the purpose of fine segmentation. In image
restoration, we propose a nonlinear PDE for regularizing a tensor which contains
the first derivative information of an image such as strength of edges and a direction
of the gradient of the image. It improves the quality of results in many low level
topics in computer vision, which need the first derivative information of an image.
In the first segmentation algorithm, noble forces based on active contours models
are proposed for capturing objects in an image. The main purpose of segmentation
is to detect multiple junctions and holes of objects in an image. Contemplating the
common functionality of forces in previous active contours models, we propose the
geometric attraction-driven flow (GADF), the binary edge function, and the binary
balloon forces to detect objects in difficult cases such as varying illumination and
complex shapes. The orientation of GADF is orthogonally aligned with a boundary
of an object and two vectors across the boundary are the opposite direction. Since
GADF is obtained robust to changes of strength of edges, it prevents a leakage on
an weak edge in curve evolution. To reduce the interference from other forces, we
design the binary edge function using the property of orientation in GADF. We also
design the binary balloon forces based on the four-color theorem. Combining with
initial dual level set functions, the proposed model captures holes in objects and
multiple junctions from different colors. The result does not depend on positions of
initial contours.
In the second segmentation algorithm, we propose fine segmentation in order to
i
extract objects in an image without loss of detailed shapes. The proposed method
is well performed for the image which has simple background colors or simple object
colors. The GADF and edge-regions are combined to detect boundaries of objects
in a sub-pixel resolution. The main strategy to segment the boundaries is to con-
struct initial curves close to objects by using edge-regions and then to make a curve
evolution in GADF. Since the initial curves are close to objects regardless of shapes,
highly non-convex shapes are naturally detected and dependence on initial curves in
boundary-based segmentation algorithms is removed. Moreover, weak boundaries
are captured because the orientation of GADF is obtained robust to changes of
strength of edges.
According to the main purpose of segmentation which is fine extraction of objects
or measurement of sizes of objects, we propose a local region competition (LRC)
algorithm. This noble algorithm detects perceptible boundaries which can be used
to extract objects from an image without visual loss of detailed shapes. The LRC
and edge-regions make distinctive difference from the first segmentation algorithm.
We have successfully accomplished fine segmentation of objects from images taken
in a studio and aphids from images of soybean leaves.
In image restoration, we propose a nonlinear partial differential equation (PDE)
for regularizing a tensor which contains the first derivative information of an image
such as strength of edges and a direction of the gradient of the image. Unlike a
typical diffusivity matrix which consists of derivatives of a tensor data, we propose
a diffusivity matrix which consists of the tensor data itself, i.e., derivatives of an
image. This allows directional smoothing for the tensor along edges which are not in
the tensor but are in the image. That is, a tensor in the proposed PDE is diffused fast
along edges of an image but slowly across them. Since we have a regularized tensor
which properly represents the first derivative information of an image, the tensor
is useful to improve the quality of image denoising, image enhancement, corner
detection, ramp preserving denoising, image inpainting, and image magnification.
We also prove the uniqueness and existence of solution to the proposed PDE.
ii
Contents
Abstract i
Contents iv
List of Tables vi
List of Figures viii
1 Introduction 1
2 Image Segmentation 7
2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Segmentation using GADF . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.1 Overview: common terms in active contours . . . . . . . . . . 9
2.2.2 Geometric attraction-driven flow . . . . . . . . . . . . . . . . 15
2.2.3 Binary edge function . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.4 Binary balloon force . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.5 Examples and numerical aspects . . . . . . . . . . . . . . . . 30
2.3 Fine segmentation using edge-regions . . . . . . . . . . . . . . . . . . 38
2.3.1 Overview: algorithms . . . . . . . . . . . . . . . . . . . . . . 38
2.3.2 Step 1: Detection of edge-regions . . . . . . . . . . . . . . . . 40
2.3.3 Step 2: Construction of initial curves for segmentation . . . . 46
2.3.4 Step 3: Placement of curves on exact boundary . . . . . . . . 51
2.3.5 Step 4: Local region competitions for perceptible boundary . 53
2.3.6 Examples and numerical aspects . . . . . . . . . . . . . . . . 55
3 Image Restoration 62
3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
iv
3.2 A nonlinear PDE for regularizing a tensor . . . . . . . . . . . . . . . 65
3.2.1 Modeling of PDE . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.2.2 Quality of the nonlinear structure tensor . . . . . . . . . . . . 68
3.2.3 Different types of PDEs . . . . . . . . . . . . . . . . . . . . . 73
3.3 Existence and uniqueness . . . . . . . . . . . . . . . . . . . . . . . . 75
3.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.4.1 Corner detection . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.4.2 Image denoising and enhancement . . . . . . . . . . . . . . . 86
3.4.3 Ramp preserving denoising . . . . . . . . . . . . . . . . . . . 93
3.4.4 Image inpainting . . . . . . . . . . . . . . . . . . . . . . . . . 96
3.4.5 Image magnification . . . . . . . . . . . . . . . . . . . . . . . 99
4 Conclusion 102
References 105
v
List of Tables
2.1 General problems in active contours for image segmentation . . . . . 8
2.2 Comparison of active contours based on (2.8). Fs is to control smooth-
ness of contours, Fb is to force contours to move from far distance
toward the boundary of objects, Fa is to attract contours much closer
to the boundary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 When the proposed active contours model (2.9) is used, we check the
success of capturing the object for different Fb and different positions
of zero level set Γφ0 of the level set function φ0. Note that, even
though Γφ0 = Γ−φ0 , they give different results. Four different posi-
tions of the initial contour Γφ0 and the regions Ω1 and Ω2 are shown
in Figure 2.3. Notice that the results of using Fb with φ0 and −Fbwith −φ0 in the Cases III and IV are exactly same. We call the pairof level set functions φ0 and −φ0 as initial dual level set functions. . 26
2.4 It shows the robustness of the proposed algorithm to different noise
levels. The Gaussian white noise is added with the zero mean and
the different values of the standard deviation σ from 10 to 100. The
accuracy is computed by using the relative length in (2.24). The
images on the right show different noise levels, from the top left to
the bottom right, σ = 20, σ = 50, σ = 70, and σ = 100. . . . . . . . 34
vi
3.1 (a) is a clean image Ic. In (b), we add the Gaussian white noise with
zero mean and the standard deviation 70 (SNR ' 7.62). (c), (d), and(e) are obtained by different PDEs for image denoising (3.13), (3.14),
and (3.15) at T1 = 30, respectively (see Section 3.2.3). From top to
bottom, different end time T2 for regularizing a tensor is used as 1,
5, and 10. SNR and relative H1 norm error are computed by (3.9)
and (3.10), respectively. Note that denoised images in (e) from (3.15)
which uses our regularized tensor (3.7) preserve geometric features
such as edges and corners in the original image (a) and have steady
and high SNR with various end time T2. . . . . . . . . . . . . . . . . 69
vii
List of Figures
1.1 The left image is a digital image. The graph in the middle is the
intensity profile of the left image. It assumes that an image is a
positive real-valued function defined on a rectangular domain even
though a digital image is represented by integers from 0 as black to
255 as white. The right image shows the symmetric and periodic
extension of the left image. . . . . . . . . . . . . . . . . . . . . . . . 2
2.1 We compare the orientation of GADF with the orientation of the
GVF and the gradient of edge function ∇g. Note that ∇g ‖ ∇fwhere f is the edge map. If two vectors have opposite direction, they
are highlighted with the yellow color in (d), (e), and (f). Clearly, the
GADF in (a) has better information than the GVF and the gradient
of edge function along the boundary of object. . . . . . . . . . . . . 19
2.2 The GADF (2.13) for an image as a real-valued function I: [0, 1] ⊂R → R+. From the top, the graphs are the image I, I ′, and I ′′.The point x0 is the edge (2.16). The sign at the bottom is obtained
by (2.17) and the arrows represent the GADF. . . . . . . . . . . . . 21
2.3 (a) is the original image and the black object is placed in the middle.
(b) shows the regions in (2.22) after the binary edge function is ob-
tained. We denote Ω1 and Ω2 as connected components of Ωb which
is the support of the binary edge function. (c) is different initial con-
tours Γφ0 . Note that the initial level set function φ0 is positive inside
the contour and negative outside the contour. . . . . . . . . . . . . . 25
viii
2.4 From top to bottom, the evolving contours of the proposed model (2.9)
are shown with different positions of the initial contour. We use dual
level set functions, φ0 and −φ0, as the initial condition and the binaryballoon force as the Case III in Table 2.3. The level set function φ0 is
positive inside the contour and negative outside the contour. The red
and blue contour are evolved from φ0 and −φ0, respectively. Noticethat, by using initial dual level set functions and the proper binary
balloon force, the proposed model has the capability of capturing the
object regardless of the position of initial contours. . . . . . . . . . 27
2.5 (a) is the original image and the object in the middle has holes and
multiple junctions. The gray region in (b) is Ωa and four colors are
labeled on connected components of Ωb in (2.22) based on the four-
color theorem. (c) and (d) are the profiles of F 1b and F2b ; white, gray,
and black represent 1, 0, and −1, respectively. The green contoursin (f) are initial contours. The contours in (g) are the result of the
proposed model (2.9) from initial dual level set functions with the
binary balloon force F 1b . The contours in (h) are obtained in the
same way from F 2b . Combining two contours in (g) and (h), the final
result in (e) is obtained. The evolving contours from (f) to (g) and
(h) are shown in the third and the fourth row, respectively . . . . . 29
2.6 The contours in (b) and (d) are the results of the proposed model (2.9)
from the initial contours in (a) and (c), respectively. The GADF
captures the weak edge which is the left part of the rectangle frame.
The RAGS in (f) also capture the weak edge when the region map
gives the correct information. In (e) and (f), we use two different
region maps to obtain the result of the RAGS. The contour from the
GAC and the GVF passes by the weak edge. Note that we capture
the whole frame in (d) by using the general type of initial contours in
(c). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
ix
2.7 The initial contour in each method is placed at the boundary of image.
The moving contours are shown from top to bottom. The illumination
is changed on the object and its background. The GADF clearly
captures the object. Note that the region map in the RAGS is almost
same as the object. The ACWE in (d) may capture the object by
manipulating four parameters in the formulation (2.6), however, it
does not work with λ1 = λ2 = 1, α = 0.01, and η = 0. The simplified
version of the GAR [48] in (e) does not capture the rectangle because
the Gaussian distribution is not adjustable to represent the varying
illumination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.8 Multiple junctions and holes are captured by the proposed model.
Note that the varying illumination on each hole is caused by the
shadow. It makes the same difficulty to capture the object as in
Figure 2.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.9 The images are taken in the photo studio. The objects are commercial
products, the first row is a part of DVD player and the others are
parts of a component in a machine. Even though the objects are
taken on the simple background, there are well-known difficulties in
image segmentation: the weak edge and complexity of shapes such as
holes and multiple junctions. Note that weak edges are shown in the
right side of DVD in the first row and the bottom of the object in the
last row. The proposed model clearly captures boundaries of objects. 36
2.10 We use some of images from [30,31]. The proposed model captures the
object in each image. The holes in the object and multiple junctions
from different colors are detected, but some of thin branch in the first
row and the left ventral fin of the bream in the third row are not
clearly segmented. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
x
2.11 The vector field in the left image is the GADF (2.13) and the blue
curve is the exact boundary obtained from the segmentation algo-
rithm in the previous section. The significant difference between the
exact boundaries and the perceptible boundaries is show on the right
diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.12 In (a), an ideal case of normalized vectors ~w/|~w| around x0 whichrepresents an edge are shown. (b) is a small part of Figure 2.23. The
vector in (c) is the normalized vector field ~w/|~w| and the black regionin (c) represents the candidates of edge-regions. . . . . . . . . . . . 42
2.13 Bad candidates and edge-regions: (a) is an original image. The black
region in (b) represents candidates of edge-regions and the candidates
in yellow areas are bad candidates which are not parts of boundaries
of objects obviously. The black regions in (c) are the edge-regions. . 44
2.14 Procedures of solving (2.34): (a) is a small part of Figure 2.23. The
black regions in (b) are edge-regions and the curves Γψ(·,0) in (b) are
the initial condition for (2.34). The curves Γψ(·,T1) in (c) are a result
of (2.34), which connects edge-regions along boundaries of objects.
The curve Γψ̃
in (d) is obtained from the curves in (c) and will be
used as an initial curve for the segmentation process. . . . . . . . . 47
2.15 Eigenvectors vλ and evolving curves ψ(x, t): The concave parts of
curves will stay because of the restriction in (2.36). The curves in (a)
are an initial condition for (2.34) and the curves in (b) are obtained
at time T1. Note that the image is a small part of Figure 2.23. . . . 48
xi
2.16 The effect of the force Fs(x) in the result of (2.34): a result without
using Fs(x) has excessive connections. The curves in (a) are the initial
condition Γψ(·,0) for (2.34). The profile in (b) is Fs(x). The curves
Γψ(·,T1) in (c) are a result in (2.34) without using Fs(x). The curves
Γψ̃
in (d) from (c) are bad initial curves for the segmentation process
since they are far from the object. The curves Γψ(·,T1) in (e) are a
result in (2.34) with Fs(x). The curves Γψ̃ in (f) from (e) are good
initial curves for the segmentation process. Note that the image is a
small part of Figure 2.23. . . . . . . . . . . . . . . . . . . . . . . . . 50
2.17 Initial curves and the result of (2.37): While the contours of simply
connected regions are disappeared, the outer contour of multiply con-
nected regions stays at exact boundaries of objects. The vectors are
GADF. A result Γψ(·,T1) from (2.34) is in (a). The image (b) shows
where simply connected regions and a multiply connected region are.
The curves Γψ̃
in (c) are initial curves Γφ(·,0) for (2.37). The curve
in (d) is a result of segmentation from (2.37). Note that the image is
a small part of Figure 2.23. . . . . . . . . . . . . . . . . . . . . . . . 52
2.18 Conceptual diagram for understanding the force F̂s in (2.38). The
curve Γϕ(·,t) is going to move inward. . . . . . . . . . . . . . . . . . . 55
2.19 The perceptible boundary, the exact boundary, and the extracted
objects: The image in (a) is a small part of Figure 2.23. In (b), the
red curve is the perceptible boundary and the blue curve is the exact
boundary. The extracted object from the perceptible boundary is
with white background in (c). The extracted object from the exact
boundary is with white background in (d). See visual loss of the
object in (d), which can be hardly seen in (c). . . . . . . . . . . . . . 56
xii
2.20 A procedure of a fine segmentation algorithm using GADF and edge-
regions: (a) is an original image. The black regions in (b) are edge-
regions. The curves in (c) are a result of (2.34). In (d), initial curves
for a segmentation process are shown. The curves in (e) are percep-
tible boundaries. The image in (f) is an extracted object on white
background. The size of image is 940 by 544. . . . . . . . . . . . . . 57
2.21 An example for 3D VR content: The yellow part is highly non-convex.
The size of image is 468 by 576. . . . . . . . . . . . . . . . . . . . . 58
2.22 An example for 3D VR content: As a smooth lighting condition is
used, the original color of shiny surface is seriously changed because
of total reflection. It generates weak edges near boundaries of the
object. The size of image is 288 by 288. . . . . . . . . . . . . . . . . 58
2.23 An example for 3D VR content: It has a repeated non-convex shape
and weak edges changed smoothly from strong edges due to a reflec-
tion of light. The size of image is 1056 by 496. . . . . . . . . . . . . 59
2.24 Segmenting aphids in a soybean leaf: Original image and segmenta-
tion of aphids. The size of each image is 640 by 480. From top to
down, it is getting old leaves. Since the color is changed depending
on the age of leave, the color extraction approach does not work. . . 61
3.1 (a) is an one-dimensional image data I(x). The point x = e is an
edge in (a). (b) is u(x, 0) =(dIdx(x)
)2. (c) is
∣∣∂u∂x(x, 0)
∣∣2. . . . . . . . . 653.2 (a) is an one-dimensional noisy image data I(x) (green) and an orig-
inal data (red). (b) is an initial data of (3.6) (green) and the square
of derivative of the original data (red). (c) is regularized derivatives
at T2 = 600 using different PDEs. The green curve is a result of the
proposed PDE (3.6) and the blue curve is a result of (3.5). . . . . . 67
xiii
3.3 (a) is a given image and the red curve in (b) is the exact location of
edges. (c) is the vector field V in (3.11) obtained by the regularizedtensor of the proposed PDE (3.7) at T2 = 10. Vectors on the set Rin (3.12) are highlighted in yellow. (d) and (e) are magnified images
from both the red curve in (b) and vectors in (c) on green square
regions. Note that vectors in highlighted in yellow are placed in pixels
close to edges. They also point to edges and are aligned orthogonally
to edges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.4 A restricted vector field V|R is obtained by (3.11) and (3.12) from atensor. Vectors in (a) are V|R from a tensor ∇Ĩ∇ĨT, where a regular-ized image Ĩ is obtained by the PM model (3.13). Vectors in (b), (c),
(d), and (e) are V|R from the regularized tensors of (3.16), (3.17), (3.18)and (3.19), respectively. From top to bottom, end time T1 for regu-
larizing an image and T2 for regularizing a tensor are taken as same
values, 11, 33, and 55. Apparently, vectors from our regularized ten-
sor (3.19) show the best result for preserving derivative information
near edges in an image as time evolves. . . . . . . . . . . . . . . . . . 72
3.5 (a) is an original image. In (b), we add the Gaussian white noise with
zero mean and the standard deviation 50 (SNR ' 14.49). From (c) to(f), they are profiles of minimum eigenvalues from different regular-
ized tensors in (3.16), (3.17), (3.18), and (3.19) at end time T2 = 70,
respectively. Images from (c-1) to (f-1) are magnified from a part on
the top right rectangle from (c) to (f) and we plot intensity graphs
of the minimum eigenvalue with the part of the image. The profile
(f-1) shows the best result which has four sharp peaks at corners and
flatter shape on homogeneous regions in the image. . . . . . . . . . . 85
xiv
3.6 The comparison of classical denoising algorithms. From the left on
top, we have the original image which is corrupted by granular noise in
the negative film, the image denoised by the proposed algorithm (3.32),
the image denoised by the median filter, and the last image denoised
by the Gaussian filter. The figures on the bottom represent intensity
plots of the corresponding gray image. We can see how the initial im-
age is corrupted. The median filter shows a typical stair case problem
and the Gaussian filter deteriorates detailed shapes. The proposed al-
gorithm gives the best result. . . . . . . . . . . . . . . . . . . . . . . 87
3.7 (a) is an original clean image. In (b), we add the Gaussian white
noise with zero mean and the standard deviation 50 (SNR ' 11.97).We use end time T2 = 5 for obtaining a diffusivity matrix in (3.33).
The result (c) is a combination of the Perona-Malik model for a color
image with a fidelity term and a shock filter. (d) is the result of
proposed method (3.33) which preserves corners and edges very well.
In the second and the third row, we magnify two parts in the first row. 88
3.8 (a) is an original image which has jpeg artifacts. In (b), we add
the Gaussian white noise with zero mean and the standard deviation
10. We use the end time T2 = 10 for obtaining a diffusivity matrix
in (3.33). (c) and (d) are obtained in the same way as in Figure 3.7-
(c) and 3.7-(d), respectively. The result in (d) from the proposed
method preserves significant features around the frame of glasses. In
the second row, we magnify a region around the right eye in the first
row. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.9 The corrupted image by bad weather and inadequate ISO in a digital
camera. It is enhanced in Figure 3.10. . . . . . . . . . . . . . . . . . 90
3.10 The enhanced image obtained by (3.33) from the corrupted image in
Figure 3.9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
xv
3.11 Images in (a) are an original clean image at the top row and an image
at the bottom row with the Gaussian white noise with zero mean
and the standard deviation 20 (SNR ' 14.56). (b) is a result of theproposed model (3.34) at T1 = 17. We use different end time T2 = 1
at the top row and T2 = 5 at the bottom row. The result preserves
ramp structure of the original image. . . . . . . . . . . . . . . . . . . 93
3.12 (a) is same images in Figure 3.11-(b). (c) is a vector field obtained
by (3.11) and (3.12) from a tensor in (3.34). The position of yellow
vector field indicates where the maximum eigenvalue of a tensor has
a local maximum. The noisy data in Figure 3.11-(a) is diffused slowly
across yellow regions in the proposed model (3.34) and ramp structure
is preserved. In (b), we show both the left of (a) and the right of (c). 94
3.13 We magnify images in Figure 3.12-(c) in order to see the direction of
vector field. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.14 We use an image from the Berkeley Image Database [31]. (a) is an
original image. The green curve in (c) is the region D contains the
damaged image and the image (d) shows the initial condition in (3.35).
The image (e) is the result at T1 = 500. In (b), we put the inpainted
image (e) into the original image to see how it looks naturally. . . . 97
3.15 The top is an original image, the middle is an initial image in (3.35),
and the bottom is recovered image. The brand name is clearly deleted. 98
3.16 (a) is original image. (b) is the magnified image from (a) in Section 3.4.5.100
xvi