Edge Detection and Active Contours - Télécom ParisTech...–Finite Differences: each element of the contour is viewed as a point with individual mechanical properties. –Finite

Elsa Angelini, Florence Tupin

Department TSI, Telecom ParisTech

[email protected]

2011

Edge Detection and Active Contours

Outline

• Introduction

• Edge Detection

• Active Contours

Introduction

• The segmentation Problem

Segmentation

Edges Regions

Textures

Measures Shape

Recognition

Structural

Scene Analysis

Introduction

• What is an edge ?

Introduction


Introduction


Introduction

• Contours profiles

Staircase Ramp Roof

Introduction

• The segmentation problem: – Partition an image into objects: – 2 approaches:

• Region-based • Contour-based

• Edge Detection: – Detection of « abrupt changes » in image

gradient. – Analysis of first and second derivatives of

image gradients.

Outline

• Introduction

• Edge Detection

• Active Contours

Edge detection

• Ideal 1D edge

Edge detection

• Ideal 1D edge

Edge detection

• Ideal 1D edge

Edge detection

• Case of a 2D image

Edge Detection

• Gradient-based

( , )I x yImage with a continuous representation:

,

( , ),

I x y

xG I x y

I x y

y

2 2

, ,( , )

I x y I x yG I x y

x y

• Gradient-based

Edge detection

• Detection of gradient modulus maxima, in gradient direction

Edge detection

• Detection of gradient modulus maxima, in gradient direction

Edge detection

1 0 1

1 0 1

1 0 1

1 1 1

0 0 0

1 1 1

Prewitt Sobel

1 0 1

2 0 2

1 0 1

1 2 1

0 0 0

1 2 1

0 0 0

0 1 0

0 1 0

0 0 0

0 1 0

0 0 1

Gradient Roberts

Oriented edges.

• Gradient-based Filters

Edge detection

• Gradient-based – Dedicated Gradient Filters

1. Pre-processing: filtering (Gaussian, Median).

2. Segmentation via thresholding or local maxima detection.

3. Post-processing: contour closing, curve fitting, smoothing.

Edge Detection

• Gradient-based

– example: Boundary Tracking

Boundary tracking is very sensitive to noise ⇒ Use of smoothing, average gradient computation, large « tracking » neighborhoods.

Edge map

Edge Detection

• Gradient-based – example: morphological post-processing

www.mathworks.com

Edge Detection

• Laplacian-based

Edge 1st derivative 2nd derivative

2 2

2 2

, ,( , )

I x y I x yI x y

x y

Zero Crossing

Edge Detection

• Laplacian-based

Edge Detection

• Laplacian-based

Edge Detection

• Laplacian-based

Laplacian operator on the image: Discrete implementation with convolution kernels:

0 1 0

1 4 1

0 1 0

1 1 1

1 8 1

1 1 1

2 convolution kernels

-Set of closed connected contours…but…

Very sensitive to noise!

Edge Detection

• Laplacian-based: Laplacian of Gaussian (LoG)

2 2

2 2

2 2

2 2

, , , ,( , )

, ,, ,

I x y G x y I x y G x yLoG I x y

x y

G x y G x yI x y I x y

x y

Convolution kernel?

Edge Detection

• Laplacian-based: Laplacian of Gaussian (LoG)

2 2 2 2

2 22 2

2 2

2 4 2

1 11

2 2

x y x y

x yG e e

Impulse response Transfer function

Band-pass

Convolution kernel

Edge Detection

• Laplacian-based

– Parameter controls the width of central peak : amount of smoothing.

- Good approximation with Difference of Gaussians (DoG), with a ratio 2 / 1= 1.6.

– DoG separable in x and y : efficient implementation.

Edge Detection

• Laplacian of a Gaussian (LoG)

Edge Detection

• Laplacian of a Gaussian (LoG)

Edge Detection

• Analytical: – CANNY:

• Hypothesis: 1D contours, staircase model, white Gaussian noise

• Edge detection via detection of local maxima of Linear Filtering.

I x AH x n x

0 0edgeO x I x f x x dx

?edgefedge edgef x f x

Edge detection

Edge Detection

0

2

'

'2

0'

''2 2

0

0

edge

edge

edge

edge

edgeedge

edge edge

f x dx

f x dx

f

f x dx

f x dxfk

f x dx f x dx

Good Detection

Good Localization

Unique Response

• Analytical – CANNY:

• Performance Criteria:

Edge Detection

• Analytical – Canny:

• Optimization of the 3 criteria with Lagrange multipliers.

' ''

2 '2 ''2

1 2 3

'' '''

1 2 3

, , ,

2 2 2 0

edge edge edge edge

edge edge edge

b

edge edge edgea

b

a

Max L x f f f x dx

f f f f dx

f f f

• Analytical – CANNY

• Solution

1 2 3 4

2 2221 1 1 2

2 2 2 2

2 2

sin cos sin cos

410; ;4

4 2 2

x x x x

edgef x a e wx a e wx a e wx a e wx

ww

' '

on , :

0 0; 0; 0 ; 0

1.12

edge edge edge edge

FIR M M

f f M f S f M

Edge Detection

• Analytical 1. CANNY‟s approximation with derivatives of Gaussian

(IIR).

2. CANNY‟s approximation with IIR (DERICHE)

2

22, 0.92

x

edgef x xe

' '

2 2

0 0; 0; 0 ; 0

sin

22 ;

edge edge edge edge

x

edge

f f f S f

f x ce wx

w

Edge Detection

• Analytical – DERICHE Deriche – gaussian

Edge Detection

• In 2D : – Derivation in one direction :

– Smoothing in the opposite direction :

– 2D expressions :

Edge Detection

• In 2D :

Edge detection

• Post-processing : – Hysteresis :

• A high threshold to select very reliable edges • A low threshold to select continuous edges • Combination of both by connex component

validation

Edge detection

• Post-processing : – Edge closing by dynamic programming

– Hough transform

Outline

• Introduction

• Edge Detection

• Active Contours

Active Contours

• Formulations: – Parametric – Geometric – Statistics – Graph-cuts

• N-D Implementation • Applications • Conclusions and Perspectives

Active Contours

• Contour placed in the data space and deforming towards an optimal position and shape.

• Forces of Deformation: – Internal forces: define intrinsic shape properties

⇨ preserve shape smoothness during deformation. – External forces: defined from the data.

⇨ Control contour deformation according to data content (e.g: edge locations).

Active Contours

• Parametric contours.

• Geometric contours.

Parametric Active Contours

Formulation of the Problem 1. Energy Minimization :

Minimize the weighted sum of internal and external energies (Force potentials).

Final contour position corresponds to an energy‟s minimum.

2. Dynamical Forces : Equilibrium between internal and external

forces at each point on the contour.


Definition of the Energy • An active contour is a curve v(s) = [x(s), y(s)], where s

∈ [0, 1] is the arc length.

• v(s) evolves towards a position minimizing the energy functional:

Etotal = Einternal(v(s)) + Eexternal(v(s))

s


Einternal(v(s)) :

Goal: Obtain a smooth contour

– Penalize the size of the object ⇨ increased energy with high area and perimeter values.

– Penalize irregular contours ⇨ minimize the contour curvature. – Constrain the shape of the contour: circles, ellipses, ...

Minimization Method: optimization.

– Finite formulation: Exhaustive search of a minimum, global or with probabilistic algorithms.

– Infinite formulation: local minimum via progressive adaptation, gradient descent or other PDE solvers.


Einternal(v(s)) :

221

0

2

2interne

v s v s

s ss dsE s

length of the contour ⇨

tension controlled by the

elasticity coefficient.

Curvature of the contour

⇨ rigidity controlled by

rigidity coefficient.

s

' ', tan ,d dy

ds x yds dx


Eexternal(v(s)) : – Standard formulation: Integral of a force

potential.

– Potential : low values on the contours in the image

(e.g.: derived from image gradient)

1

0externeE P v s ds

2

, ,P x y w I x y

2

, , * ,P x y w G x y I x y


Energy Minimization • Goal: find the contour v(s) that minimizes the global

energy.

• Framework: Attract an initial contours towards contours of the image, while avoiding stretching and bending.

• Method 1: Variational problem formulated with the Euler-Lagrange equation:

22

2 20

v s v sP v

s s s s


Energy Minimization • Method 2: Dynamic Deformable model:

– Minimization viewed as a static problem. – Build a dynamic system that we evolve towards

an equilibrium state according to a Lagrangian mechanical point of view.

– This dynamical model unifies the shape and motion descriptions, defining an active contour⇨ quantification of a shape evolution through time v(s,t).


Energy Minimization • Method 2: Dynamic Deformable model:

– Motion equation: according to the 2nd law of Newton, :

– Equilibrium state:

22 2

2 2 2

v sv v vP v

t t s s s s

2

20

v v

t t


Energy Minimization Methods 1 & 2: no analytical solution (due to

external energy). ⇨ Need to discretize :

– Finite Differences: each element of the contour is viewed as a point with individual mechanical properties.

– Finite Elements : sub-elements between nodes.

• N control points v = (v1, v2, . . . , vN), distant with a spatial step h.


Energy Minimization: Numerical Schemes • Discretization of spatial derivatives • Matrix Notation ( cst): Penta diagonal matrix

2

1 1

2 2

2i i i iv v v v

s h

4

2 1 1 2

4 4

4 6 4i i i i i iv v v v v v

s h

Av P v

2 2 2

2 2 2 2

22 2

2

2 6 4 0 ... ...

4 2 6 4 0 ...

1

4 ...

0 ...

... ...

h h h

h h h h

Ah

h h

h

2 4

2 40

v s v sP v

s s


Energy Minimization: Numerical Schemes • Problems

– Non-linear terms in the potential force. – Matrix A non-invertible.

• Need for iterative numerical schemes

1 1 1

1 1 1

,

,

n n n n n

x

n n n n n

y

Ax P x y x x

Ay P x y y y

Inertia

coefficient

,, ,

,

x

y

Ax P x yv x y

Ay P x y

Internal forces


Energy Minimization: Parameters • Spatial continuity ⇨ spatial step smaller

than pixel size.

• = elasticity (i.e. dof of points to move away from each others).

• β = rigidity.

• Temporal step set to have a maximum displacement of 1-2 pixels at each iteration.


Energy Minimization: Conclusions Advantages • Extraction of a locally optimal position via iterative

deformation of a curve. • Suited for contour extraction:

– open, – closed (v0 = vN) – With fixed extremities (v0 et vN fixed)

• General Framework : several different types exist. • Simple an efficient 2D implementation. • Numerical stability wrt internal forces.


Energy Minimization: Conclusions

Limitations • Instability wrt external forces: if spatial step too

big, can miss some contours.

• Sensitive to local minima problems and initialization.

• Difficult parameterization.

• No change in topology allowed (i.e. division/fusion of objects).

• No simultaneous deformation of multiple objects.


Formulation of the Problem 1. Energy Minimization :

Minimize the weighted sum of internal and external energies (Force potentials).

Final contour position corresponds to an energy‟s minimum.

2. Dynamical Forces : Equilibrium between internal and external

forces at each point on the contour.

Formulation with Dynamical Forces • Dynamical problem with more general forces than

potential forces.

• Newton‟s law :


2

internal external viscous2

CF C F C F C

t

mass C

t

Formulation with Dynamical Forces • Simplification: no mass


internal external

CF C F C

t

idem Superposition

of forces

Fexternal : GRADIENT forces

• Properties of the Gradient Vectors: – Point towards the contours (normals). – Large modules near edges. – Module around 0 in homogeneous regions.

• Problems: – Weak attraction range (only close to edges) – No force in homogeneous areas (nothing moves…).

• To solve : – Initialization problems. – Convergence towards concave regions.


Example Fexternal: Balloon [Cohen et Cohen] :

• A gradient fore to attract the contours towards edges.

• A pressure force to constrain the model to inflate/deflate:

• Computational cost: image gradient and normals on the contours for each node.

• Need to control the dynamical behavior of the contour far from the edges ( weight of k1 ).


externe 1

P CF C k k n s

P Cn

+/-: inflate

Example Fexterne: Potential Forces for Distances • Distance map D(x, y) (Euclidian or Chamfer) for each

pixel to the closest point on the contour ⇨ field of potential forces.

• D(x, y) defines the potential energy… :

• … and the field of forces

• Problem remaining: concave shapes


2,

distance ,D x y

P x y we

externe distance ,F C P x y

,D x y


Example of Fexterne: GVF [Xu et Prince 1998]

• Vector field.

• Preserve gradient properties near the edges.

• Diffuse these properties in homogeneous regions via « gradient diffuse ».

Example of Fexterne: GVF • GVF is a vector field :

• is defined via energy minimization:

= Edge map of the image


, ( , ) ( , )V x y u x y v x y

,V x y

edge ,x y

2 2 2 2

2 2

edge edge

, , , ,

| , | | , , |

x y x yu x y u x y v x y v x yE dxdy

x y V x y x y

Data terms

Regularization on the overall domain

,I x yedge ,x y

,I x y

Example of Fexterne: GVF • The GVF vector field is obtained by solving the

Euler equations :


2 2

edge edge edge

2 2

edge edge edge

, , ,, , 0

, , ,, , 0

I x y I x y I x yu x y u x y

x x y

I x y I x y I x yv x y v x y

y x y

Gradient of the Edge map

Laplace Eq. Gradient data term


Final result, iter = 50GVF force



GVF


Example of Fexterne: GVF

• Numerical implementation in the computer lab…


Example of Fexterne: GVF

Bibliography

1. Kass M, Witkin A and Terzopoulos D. „‟Snakes: Active

contour models‟‟. International Journal of Computer Vision 1987; 1-: 321-331.

2. Cohen LD and Cohen I. „‟Finite-elements methods for active contour models and balloons for 2-D and 3-D Images‟‟. IEEE Transactions on Pattern Analysis and Machine Intelligence 1993; 15-11: 1131-1147.

3. Xu C and Prince JL. „‟Snakes, shapes and gradient vector Flow‟‟. IEEE Transactions on Image Processing 1998: 359-369.


Active Contours

• Parametric contours.

• Geometric contours.

Geometric Active Contours

Introduction • Theory of curve evolution and geometrical flows.

• The contour deforms with a speed made of 2 terms:

– Regularizing term (curvature-based motion).

– Expansion term or contraction to go towards image edges.

• The active contour is defined via a geometrical flow (PDE).

⇨ the curve evolution must stop at locations of image edges

corresponding to the object to segment.


• Geometric Active contours – Numerical methods via level sets.

– Geodesic.

– Mumford-Shah.


Curve Evolution Theory: • Curve evolution through geometric measures (normal vectors to

the curve, curvature, ...) and independent of curve parameterization (e.g. derivatives).

• Let a curve with spatial parameter s and temporal parameter t

• The curve evolution in the normal directions controlled by the PDE:

, ( , ) ( , )X s t x s t y s t

,X s tV N

t

NsGeometric measure of

the curve Propagation speed


Curve Evolution Theory • 1. Constant speed:

⇨ similar to a pression force (balloon).

• 2. Motion under curvature:

⇨ similar to an elasticity force.

0

,X s tV N

t

,, 0

X s tN

t


Numerical Methods with Level Sets • Goal:

– Numerical methods to compute the spatial propagation of a curve in time: add a temporal dimension.

– Precise characterization of the geometric properties of the contour.

• Approach: – Define a spatio-temporal function.

– Immerge the contour in a function as its 0-level (iso-contours).

– Extend the function to other levels.

0 ,0 0

, 0t

:


Numerical Methods with Level Sets

• Definition of the level set function: , , 0, ,

, , 0, ,

, , 0, , ,

i

o

x y t x y

x y t x y

x y t x y s t

i

o

,s t

,s t

,s t

, ,x y t

v

– is defined as the 0 level of . – deforms with a speed applied on

each point.

⇨ How to control the level set motion?

1. Define a field of speed vectors (cf. theory of curve

evolution).

2. Compute initial values of the level set function, based on the initial position of the contour to evolve.

3. Adjust the function in time, so that the level 0 corresponds to the segmentation solution.

⇨ Evolution equation for the level set function?



• Iterative Deformation Scheme:

,, 0

,, . 0

d tt cste

dt

tt

t t


• Evolution equation of the level set function:

V Nt

• Curve evolution theory:

,s t

,N s t


,

,

tN

t


• Normal vector:

,, . 0

tV t N

t

,s t

,N s t




0

,, 0

,0

tV t

t

given

,s t

,N s t




2 2

3 22 2

,

,

, 2.

,

xx y x y xy yy x

x y

tN

t

t

t

• Geometrical properties of the level set curve: directly computed on the level set function!

,s t

,N s t



,, 1

,

N tt

t

• What type of function?: – Most common choice: signed distance function

– Careful!

The solution of is not the signed distance

function of the curve solution to: .

,s t

,N s t

Vt

V Nt



• What speed of propagation ? – Take into account:

• Image Information: zero on edges from the objets to segment.

• Geometric Information of the contour: smoothing via constraints on the countour.

– Particular case: Motion under curvature

• Each part of the model evolves in the normal direction, with a speed proportional to the curvature. ⇨ points can move inward or outward, depending on the curvature‟s sign.



• Speed of propagation ? – Defined only on the contour only. – Extension of the speed over the whole domain :

• Natural Extension: e.g. with motion under curvature, computing the curvature on the overall level set function.

• Values of the nearest point from the 0-level. ⇨ problems since the speed does not depend on f and we are not anymore in the H-J framework.

, ,, , ,

x y tV x y x y t

t

,, 0

tV t

t


Reinitialization

• Why reinitialize ? – Unique correspondence between a curve and its level set function

(convergence iif convergence)

– Preserve a constant gradient norm ⇨ numerical stability.

• Methods :

– Direct evaluation:

• Detect the 0-level and re-compute the signed distance function.

• High computational cost!

– Iterative: • Equilibrium state: norm of gradient =1.

• Opposite flows for negative and positive values.

• Problem: the 0-level can move during reinitialization!


1t sign


• Narrow Band – Only evolve level sets in a narrow band around

the level zero.

– Reduce computational cost.

– No need to compute evolution speed far from the 0-level.

– less constraints on t for the CFL stability, limiting the maximum speed of deformation.



Advantages

• Change of topology

• Intrinsic geometric properties easy to compute (normals, curvatures).

• Extension to N-D straightforward : add new spatial variables to the evolution equation of the volume (x, y, z,…, t).

• Numerical implementation:

– Discretization of (x, y, t) on regular grid (x, y).

– Standard numerical schemes for the spatial derivative.


Level Sets

Limitations

3 limitations related to the numerical implementation:

• Construction of an initial level set function (x, y, t = 0) from an initial 0-level (initial contour).

• Evolution equation only defined for the 0-level ⇨ the speed function V is not defined in general for the other levels ⇨

arbitrary spatial extension.

• Instable definition of the normal to the level set function (x, y, t

= 0) ⇨ reinitialization + smoothing .


Level Sets



– Geodesic.

– Mumford-Shah.

Geodesic Deformable Models [Caselles, Kimmel, Sapiro 1997]

• Geodesic curves in a Riemannian space


• Novel approach (equivalent)

Geodesic: path (locally) minimal between 2 points.

Space with metrics defined from geodesics.

21 1

0 0

1

0

Min '

Min '

E C C s ds g I C s ds

g I C s C s dsGeodesic computation

• Euclidian Geodesics


Motion under curvature provides fastest minimization of LE.

• Geodesics for deformable models

ELength L 'C C s ds dS

Motion under curvature = NC

tEuclidian Metric and

curvature

S

1

R0

0

Lenght L '

'EL C

C g I C s C s ds

g I C s C s dS

Image characteristics

=g I N- g.NC

Nt

Geodesic Deformable Models



– Geodesic.

– Mumford-Shah.


• Variational method.

• Image I0 segmentation, defined on the domain is provided by a pair with: C contours in the image and I a smooth approximation of I0

• Energie associated with the segmentation:

,C I

2 2

0\ \

, lengthC C

E C I I d C I I d

1D, # of points on C 2D, perimeter of C 3D, surface of C

Mumford and Shah


• Conjuncture – There exists a minimal segmentation made of a finite set of

curves C1.

– [Morel-Solimini 1995, Aubert-Kornprobst 2000]: – There exists a minimal segmentation.

– The minimal segmentation is not unique.

– The ensemble of solutions is a compact set.

– Contours are rectifiable (i.e. of finite length).

– All contours can be included in a single rectifiable curve.

Mumford and Shah

22

1 2 1 2, ,

0

inside C outside CE c c C Length C I c I c

I

• Particular case: – I0 is a cartoon-like image.

– The smooth approximation I of I0 is a piecewise constant image with values c1 et c2 which are the mean values of I0 in the object and the background.

– The contour C corresponds to the contours of the objects.


Mumford and Shah

c1 = mean value inside C

c2 = mean value outside C

L(C) = length of C

A(C) = area of C

22

1 0 1 2 0 2inside C outside C

E C L C A C I c d I c d

Regularizing terms

(cf. internal energy)

Homogeneity constraints

(cf. external energy)

Deformable models without Edges [Chan, Vese, IEEE TIP 2001]


Mumford and Shah

( ){ }

( ){ }

( ){ }

/ 0

/ 0

/ 0

N

N

inside

N

outside

C x x

C x x

C x x

f

f

f

= Î =

= Î <

= Î >

¡

¡

¡

1. Insert the n-D curve C in a level

set function (n+1)-D

2. Define a Heaviside function H( ) ( )1 0

0 0

if zH z

if z

í £ïï= ìï ³ïî

( )( )dH z

zdz

d =3. Define a Dirac function ( )

22

1 0 1 2 0 2inside C outside C

E C L C A C I c d I c d



Mumford and Shah

0Length C Length

H d

d

0Area C Area

H d

- Heaviside Function H

1 21 arctan

2H

, , / , , 0C x y z x y z- Level set Function

= distance to the 0-level

- Dirac Function

2 2

1



Mumford and Shah

2

0 1 0 0 0 1 0 1, , 1E c c dx H I c H dx I c H dx

0 10 1

, ,Segmentation via inf , ,

c cE c c

( ) ( ) ( )

( )

e

e

fd f m n l l

f

d f f

f

é ùæ öÑ ÷çê ú÷- - - + - =ç ÷ê úç ÷÷ç Ñè øê úë û

¶= ¶W

Ñ ¶

2 2

0 0 1 1 0

0 ,

div I c I c

onn

length

area homogeneity homogeneity

( )( ) ( )( )

( )( )

( )( ) ( )( )( )

( )( )( )

0

1

, , , ,

, ,

, , 1 , ,

1 , ,

I x y z H x y z dxdydzc

H x y z dxdydz

I x y z H x y z dxdydzc

H x y z dxdydz

ff

f

ff

f

W

W

W

W

=

-=

-

ò

ò

ò

ò



Mumford and Shah


Bibliography

– D. Mumford and J. Shah, "Boundary detection by minimizing functional," International Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, pp. 22-26, 1985.

– S. Osher and J. A. Sethian, "Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations," Journal of Computational Physics, vol. 79, No. 1, pp. 12-49, 1988.

– V. Caselles, F. Catté, B. Coll, F. Dibos, ”A geometric model for edge detection”, Num. Mathematik, 66, 1-31, 1993.

– R.Malladi, J.A. Sethian, Baba C. Vemuri: “Shape modeling with front propagation: A level set approach ‟'. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(2): 158-175, 1995.

– T. F. Chan and L. A. Vese, "Active contours without edges," IEEE Transactions on Image Processing, vol. 10, No. 2, pp. 266 - 277, 2001.

Active Contours

Two families of N-D Active Contours: • Parametric :

– Explicit representation of the contour. Compact representation allowing fast implementation (enable real-time applications). Changes of topology very difficult to handle (in 3D).

• Geometric:

– Implicit representation of the contour as the level 0 of a scalar function of dimension (N-D+1). Contour parameterization after the deformations. Flexible adaptation of the contours‟ topology. Increase dimension of space search.

Documents

Edge Detection and Active Contours - Télécom ParisTech...–Finite Differences: each element of the contour is viewed as a point with individual mechanical properties. –Finite