Variational Principles in Optical Flow Estimation and Tracking

13

Shape Analysis towards Model-basedSegmentation

Nikos Paragios and Mikael Rousson

Abstract

In this chapter, we investigate shape modeling and registration towardsmodel-based shape-driven object extraction and segmentation. The chapterconsists of two major contributions: (i) a variational level set approach forglobal-to-local shape registration and (ii) an energetic formulation to imposeprior shape knowledge within implicit representations. Distance transformsare proven to be a very efficient feature space to perform shape registration.Prior knowledge is an important tool in the segmentation process. Followingthe example of shape registration, we introduce a stochastic level set priorthat constrains the segmentation result to be in a family of shapes definedthrough a similarity transformation of the prior model. Promising results andsystematic validation for each of the topics under investigation demonstratethe performance and the potentials of our approach.

13.1 Introduction

Shape analysis is a particularly interesting area of study in computer vision. Theapplication domain is wide and includes pattern recognition, (object recognition)medical imaging (registration), image processing (segmentation), etc. where theanalysis of shapes can provide strong support to image understanding and all itspossible applications. Two important issues in shape analysis are shape modelingand shape registration.

Shape modeling can be stated as follows: given a selection of points in 2Dor 3D space, find an appropriate representation that can describe efficiently theunderlying structure from which these points originate. Limiting the number ofparameters as well as efficiently estimating them are key objectives of theserepresentations. Additionally, invariant shape representations (according to someparticular transformations) are very attractive to imaging and vision applications.

232 Paragios & Rousson

Parametric curves and surfaces, volumes, or point clouds are some of the rep-resentation that can be found in the literature. Applications of shape modelingare numerous. In medical image analysis for example, one can consider the useof organ shape models for patient-drivent simulations. Registration between themodel and the observed structures is then required. Recognition of faces is an-other example that requires proper modeling of the face using shape as well asvisual descriptors.

Shape registration is a task under heavy consideration in shape recognition andmedical image analysis. A general registration formulation can be stated as fol-lows: given two shapes, an input� and a target�, a set of possible transformationsand a dissimilarity measure, find the transformation between � and � that mini-mizes the dissimilarity measure between the transformed model shape �� and thetarget shape �. This dissimilarity can be defined either along the boundaries (con-tours/surfaces/etc.) (shape-based) or in the entire region (area-based) enclosedby them. Application in medical imaging, motion analysis, etc. can be treatedas image/region registration/segmentation problems. Furthermore, shape registra-tion can be used as basis to shape recognition where the objective is to find from agiven set of examples the shape that provides the lower dissimilarity or the highersimilarity measurement with the target. Knowledge-based image segmentationcan be considered as step further from shape registration and is one of the mostprominent applications of shape analysis.

Using shape prior constraints to improve segmentation performance is a com-mon technique in computer vision. Dealing with physically corrupted data,occlusions and sensor noise are some of the most common problems when try-ing to segment real data. In many applications the objective is to recover somespecific objects of interest with known geometric shape properties. Towards thisend, shape models can be used to enhance segmentation performance. Shaperepresentation is the most important factor within this process and has to bechosen in conjunction with the optimization framework that is used to solve thesegmentation problem.

The reminder of this chapter is organized as follows. In Section 2 we proposean invariant to translation, rotation and local deformations shape representationwith strong discrimination power. This representation is used in section 3 withina robust statistical framework to provide an elegant solution to the problem ofshape registration. Furthermore, the modeling component as well as the regis-tration component are combined efficiently in Section 4 to introduce global shapeprior constraints in the segmentation process. Discussion is part of the last section.

13.2 Shape Modeling

A core component within the task of the analysis of shapes is the underlying shaperepresentation. The selection of this representation is a crucial aspect towards the

13. Shape Analysis towards Model-based Segmentation 233

efficient analysis of shapes within several computer vision applications such asregistration, recognition, segmentation, etc.

Point clouds is the simplest way to represent shapes in the 2D or the 3D space.Distance maps [442], snake models with elastic properties [262], Fourier de-scriptors, deformable models/templates [50], active shapes [135], shock graphs[465, 483] medial axis/skeletons [612] are representations that can be recoveredfrom the literature.

Although, some of these representations are powerful enough to capture a cer-tain number of local deformations, they require a large number of parametersto deal with important shape deformations, and they cannot deal with changesof topology. An emerging way to represent shapes can be derived using level setrepresentations [401]. This selection is invariant to translation and rotation. Underthese assumptions, given a shape �, one can define the following representation1:

��

��

��

��

��

(13.1)

where �� is the region defined by the shape and �� is the minimumEuclidean distance between the image location �� and the cloud of points �.

Distance maps, have some nice properties that describe shape in a powerfulway; they refer to structures of higher dimension where the information spacerefers to clones of the original shape positioned coherently in the image plane(iso-contours). Furthermore such representations can account for local deforma-tions that are not visible for iso-contours that far away from the original shape.Shape geometrical properties can be naturally derived from these representations,while they refer to smooth and ”monotonic” information space, with partiallycontinuous derivatives, a desirable property for most of the existing optimizationframeworks.

Shape representation is a complex procedure. In the most general case, onewould like to create a compact model that can represent a family of similar ob-jects or the same object under various conditions and view points. One way toapproach the problem is by considering a set of training examples and seekingfor a common representation that can best express the training set. Given a train-ing set of � registered shapes2 and their representations �� wewould like to recover a representation �� that can account for all examples in thetraining set while being a level set representation using distance maps as embed-ding function. Towards this end, we consider a stochastic framework [416], where

1To simplify the notation, the 2D case will be considered when introducing the differentcomponents of shape analysis.

2Shape registration will be considered in the next section and therefore the registration assumptionconsidered in the modeling phase is plausible.


the underlying shape representation refers to a stochastic level set �� ;

��

��

��

��

�� (13.2)

where �� is the shape level set. If we assume that for every given pixel �� the best value to represent the family of shapes in the training set is �� ,then the selected representation can account for local variations. Gaussian densityfunctions are used pixel-wise to describe the deviation between the selected levelset �� and the training values set at this particular location

��

Estimating the parameters of such representation is not trivial. Given regis-tered training examples one can recover the model parameters using the maximumlikelihood principle. Thus for each pixel of the image, a tuple of values is avail-able; the values of the training representations on this particular location. One canignore the specific application constraints (the overall representation has to be adistance map) and then use the maximum likelihood principle to solve the infer-ence problem, find the best estimates for �� and �� pixel-wise. Towards this end,the use of [-log] function can be used;

��

�

��

��

��

��

� �

��

��

��

��

��

��

(13.3)

where we do not impose the distance transform constraint on the form of therepresentation �� to be recovered. Furthermore, one can assume that the field�� is locally smooth. Due to the absence of data support during the estimation ofthe shape representation (limited number of samples per pixel), such smoothnessconstraints is a standard component when solving the inference problem;

��

��

��

��

��

��

��

� �

��

� ��

��

��

� �

�

��

��

��

(13.4)

where � is a blending parameter. The latest objective function does not accountfor the distance transform constraint. One can consider the use of Lagrange mul-tipliers to impose such a constraint. In this particular case such selection willnot be helpful due to the form of the cost function and the significant number ofthe unknown variables. If such constraint is not considered, one can optimize the


objective function by using the calculus of variations;

�

��

��

��

��

��

�

��

��

��

��

��

��

��

��

�

��

��

��

� (13.5)

The steady-state solution of the above motion equations can be used to deter-mine the best model parameters using the set of training examples. Such solutioncan account for optimal data support from the training set but cannot respect theconstraint of being a distance transform (��).

Optimization problems refering to solutions that are part of a constrained man-ifold are common in imaging and vision. Definition of an objective functionconstrained within this manifold is the optimal solution that can be considered.Such techniques are not always tractable when large number of unknown variablesare considered. One can consider to decouple the problem into two stages. Withinthe first stage (fitting) the objective is to find the appropriate solution with max-imal data support without considering the specific constraints. The second stageaims at finding the projection of this solution to the desired manifold. These twosteps can alterate until a steady-state solution is obtained (i.e. bundle adjustment).

Projection of the current solution of the level set representation of the model(��) to the space of distance transforms can be considered within our approach.Once an appropriate projection of the solution is obtained, then it can be usedas a starting point in the first (fitting) stage of the new iteration; estimation ofthe model parameters that best account for the shape information provided by thetraining data.

Given a shape form (zero-level set), several methods to obtain a representationthat is a distance map of this shape can be found in the literature. Our approachhas to deal with a more challenging task since the objective is to obtain a distancetransform from an image that refers to a level set representation. The extractionof the shape using zero-crossings first and then the creation of a distance map isnot an proper solution since the boundaries of the shape can be displaced unlesssuper-resolution of fine sub-pixel accuracy is considered. In order to overcomethis limitation, in [501] a partial differential equation that performs this task -without altering the boundaries - was proposed;

�

��

�� (13.6)

where ��

�is the start point of the evolution process. One can easily alternate

between the equation. (13.5) and equation. (13.6) to recover a solution that bestexpresses the training set while being a distance transform.

The obtained shape representation has all the advantages of distance transformsand can account for local variability. The recovery of the variability field [��]leads to a natural way to separate the rigid from the non-rigid parts for a givenfamily of shapes. Such property can be used accordingly to improve performance


of shape analysis techniques like registration where the shape sub-componentscan be considered in a quality-driven manner. Modeling techniques with someconceptual similarities with our shape representation can be found in [564].

13.3 Shape Registration

Shape registration has been approached into a number of ways [328]. Classifica-tion of these methods can be done according to several criteria like: (i) nature oftransformation, (ii) domain of transformation and (iii) optimization procedure.

The underlying motion model (nature of transformation) that is used to map thecurrent shape to the target is a critical component that can have an important im-pact on the performance of the registration procedure. The simplest model refersto rigid transformations (translation, rotation and scale), a compromise betweenlow complexity and fairly acceptable matching between the different structureswhen they present a dominant rigid component. Affine transformations are quitepopular and a step further due to their invariance to a variety of motions. Projec-tive motion models as well as elastic methods have been also considered to matchshapes. Regarding the transformation domain one can separate global transforma-tions (valid for the entire shape) and local ones (pixel-wise deformation models).The selection of a mathematical framework to provide the solution by meansof finding an optimum of some functional defined on the registration parameterspace is another classification criterion. These functionals attempt to quantify thesimilarity between the two shapes and can be based either in variational [105], orstochastic principles [555].

Euclidean distance transforms [442] is the most closely related representationwith the one adopted for shapes by our approach. They have been consideredmainly for image registrations in the past [126, 197, 294, 304, 547]. Comparisonbetween these methods and the proposed framework can be found at [416]. Theobjective of our work is to provide a robust global-to-local solution to the regis-tration problem. Towards this end, we introduce an optimization criterion that canaccount for global (rigid) and local pixel-wise deformations. This criterion is de-fined in the space of signed distance transforms, and is optimized using a gradientdescent method. Global and local registration parameters are recovered simulta-neously with emphasis on the global model. In order to facilitate the presentationof our approach we will consider a rigid transformation � � �� for theglobal motion model that consists of a translation component � � �� , arotation angle � and a scale parameter �.

Level set shape representations can augment the potentials on the registrationtask. A higher dimension is to be considered, where the objective is to recovera transformation � that creates pixel-wise intensity correspondences (level setscan be seen as intensity images) between the current shape representation ��and the target shape �� . We can easily prove that the selected representations[��] are invariant to translation and rotation [416, 417]. Dealing with scale


Figure 13.1. Shape registration using level set representations and global rigid models.Evolution of the registration process according to the projection of the source shape to thetarget shape using the estimates of the motion model; presented in a raster-scan format.

variations is not a standard property of these representations and requires someparticular handling. One can easily show that by up-scaling or down-scaling thelevel set representations according to the global scale registration parameter, rigidinvariant representations can be recovered;

� � ��

��

��

��

��

�cos� sin�

�sin� cos�

��

�

��

��

��

�

��

(13.7)

Registration of shapes in the 2D plane can be viewed as a global optimizationproblem involving all pixels in the image plane. Several techniques can be usedto recover the unknown transformation parameters like the sum of squared differ-ences, robust estimators, optimization of the correlation ratio, mutual information,etc. In order to introduce and demonstrate the performance of our method, at thevery beginning we will consider the simplest possible criterion, the sum of squaredifferences.

13.3.1 Sum of Square Differences

Shape registration of contours on the level set space can be considered as an im-age registration problem, a widely explored application, where the objective isto recover a global parametric transformation that creates intensity-driven corre-spondences with a minimal error. The sum of square differences can be used tomeasure the quality of the motion map;

��

��

��

��

(13.8)

where � is the residual term and the scale factor � appears in the objectivefunction to cope with scale variations. Registration is done in an augmentedshape-driven (level set) space that is robust to very local deformations and miss-ing data since the selected representation is obtained through a global procedure(Euclidean distance). Moreover, the proposed framework is invariant to rigid


(i)

Cost function

’plot1’

-50 -40 -30 -20 -10 0 10 20 30 40 -50-40

-30-20

-100

1020

3040

0

50

100

150

200

X Translation

Y Translation

Cost function

’plot1’

-50 -40 -30 -20 -10 0 10 20 30 40 -50-40

-30-20

-100

1020

3040

0

50

100

150

200

X Translation

Y Translation

Cost function

’plot1’

-50 -40 -30 -20 -10 0 10 20 30 40 -50-40

-30-20

-100

1020

3040

0

50

100

150

200

X Translation

Y Translation

Cost function

’plot2’

-1-0.5

00.5

1 -50-40

-30-20

-100

1020

3040

0

50

100

150

200

X Translation

Y Translation

Cost function

’plot1’

-50 -40 -30 -20 -10 0 10 20 30 40 -50-40

-30-20

-100

1020

3040

0

50

100

150

200

X Translation

Y Translation

Cost function

’plot4’

-50 -40 -30 -20 -10 0 10 20 30 40 -1

-0.5

0

0.5

1

102030405060708090

100110120

X Translation

Rotation

(ii)

Cost function

’plot2’

-50 -40 -30 -20 -10 0 10 20 30 40 0.70.8

0.91

1.11.20

50

100

150

200

X Translation

Scale

Cost function

’plot3’

0.75 0.8 0.85 0.9 0.95 1 1.05 1.1 1.15 1.2 -1

-0.5

0

0.5

110

20

30

40

50

60

70

80

Scale

Rotation

(1) (2)

Figure 13.2. Empirical evaluation of the cost function: (i.1) Unknown translation �� ,(i.2) Unknown transaltion �� and rotation, (ii.1) Unknown translation �� and scale, (ii.2)Unknown scale and rotation.

transformations and involves multiple matching of shapes (isophotes) that areclones of the original ones to be registered. Using the proposed formulationwe were able to convert a geometry driven point-correspondence problem intoan image-registration application where space as well feature-based (intensity)correspondences are considered.

One can consider a gradient descent method to find the minimum of the ob-jective function figure (13.1). Convexity of the objective function is a desirableproperty that can lead to a unique global solution and can be recovered by a gra-dient descent method. Distance transforms have a certain number of desirableproperties that can provide some support towards this property. They are smoothand have partially continuous derivatives. A theoretical proof regarding the con-vexity of the objective function is not trivial, and therefore one can consider anexperimental validation.

Towards this end, we have considered the shapes shown in figure (13.1). Eval-uation of the objective function in the 4D optimization space is rather impossible.In order to perform some empirical validation, one can consider several 2D sub-optimization spaces; (i) fixed scale & rotation, unknown translation, (ii) fixedtranslation, unknown rotation & scale, (iii) fixed rotation & vertical translation,unknown scale & horizontal translation and (iv) fixed rotation & horizontal trans-lation, unknown scale & vertical translation. Then, for validation purposes, onecan quantize the search space (uniform sampling of �� elements are used) for


(i)

(ii)(1) (2) (3) (4)

Figure 13.3. Global-registration for different shape categories using level set represen-tations and global rigid models. (i) Initial Conditions (source: gray, target:dark) , (ii)Registration result according to the projection of source shape to the target shape usingthe recovered motion parameters of the rigid model.

the translation between ��, the rotation between ��

��

�� and the scale

space between �� and evaluate the objective function as shown in figure(13.2). Then as shown from the experiments, form of the objective function cansupport the convex hypothesis.

Gradient descent is a standard optimization technique given the form of theobjective function leading to the following partial differential equations;

�

��

��

��

�

��

��

� ��

�

��

��

��

��

��

��

(13.9)

The performance of the proposed module is shown in figure (13.3).

13.3.2 Robust Estimators

Robustness is a critical component to registration algorithms. In order to recoverun-biased estimates using least-squares methods, numerous hard assumptionshave to be met. In the literature one can recover studies showing the instabilityof least squares estimators when a very small number of outliers is present [246].In our case due to the selected shape representation, the rich information spaceand the data support this problem was not observed unless extreme initial con-ditions are considered (experimental validation of the convexity property of theobjective function). However, local non-rigid deformations can produce outliers.


Robust estimators is an alternative to the least-squares ones and can deal withperturbed data.

M-estimators is a well known technique that can deal with outliers. Towardsthis end, the squared residuals are replaced by another function;

��

��

� �� (13.10)

where � is a symmetric and positive-definite function with a unique minimumat zero. One then can define the following influence ��

�� and weight

function��

. Some constraints are to be met by M-estimators; bounds onthe influence function, uniqueness that reflects to convex functions and non-zerogradient.

A gradient descent method can be also used to recover the solution fromobjective function leading to:

��

��

��

��

��

��

��

��

��

��

��

��

��

(13.11)

The selection of the influence function is critical. Several alternatives can befound in the literature. We have considered two well known functions, the fairand the Cauchy given by

��

��

��

��

�

��

��

��

��

��

��(13.12)

The fair function has continuous derivatives up to third order and can provideunique solution while Cauchy function does not guarantee a unique solution andhas the tendency to yield erroneous solutions due to the form of its first derivative.However such selection can eliminate to a significant factor the influence of largeerrors. One can consider the combination of these two functions [246]; using theconvex one (fair) until convergence and then apply the Cauchy to eliminate largeerrors.

One can now interpet the robust registration flows presented in equation(13.11). The influence function can be used to decrease the effect of outliers. Inparticular, modulo the selection of the function parameter one can recover a formfor the weight function that can ignore the use of ourliers during the estimationprocess.


13.3.3 Global-to-Local Registration

Global registration has limited applicability when non-rigid shapes are consid-ered. In that case global methods will be able to recover the global motion of thestructure but will fail to capture the motion details. Stability and robustness arethe main characteristics of global methods while accuracy and precise registrationcan be the outcome of local methods. The integration of global motion modelswith local deformation fields is a more appropriate way to tackle the registrationproblem.

Towards this end, we assume that the observed shape is a rigid transformation� of the target combined with some local deformations �� defined in the pixellevel for the shape components that do not follow the global motion model. Thisassumption can lead to following condition between the level set representationsof the source and the target shape;

�� Æ

��

��

� ��

�

�

��

��

��

��

��

�

��

� ��

�

��

�

��

�

��

�

��

��

��

(13.13)

where �� Æ are blending parameters. The minimization of this functional isdone using a gradient descend approach and the calculus of variations;

�

��

��

��

��

�

��

��

�

��

��

��

�

��

�

�

��

��

��

��

��

��

��

�

��

��

��

��

��

��

�

��

��

��

��

��

�� Æ ��

� �

��

��

��

��

��

��

(13.14)


(1)

(2)

(3)

Figure 13.4. (1) Global �� , � � ��, �� , (2)Global-to-Local �� , � � ��, �� , (3) LocalRegistration with regularization constraints.

13.3.4 Experimental Results & Validation

Some results on the performance of the complete system are shown in figures(13.3,13.4,13.5). In order to validate our approach, we have considered the fol-lowing experiments; (i) recover known registration parameters, (ii) validate theperformance of the method to missing and occluded parts when registration pa-rameters are known. Towards this direction, we have used two sets of real shapesfigures (13.1,13.3.3) where the source was globally deformed using a set of knowntransformations generated according to a random process in the four dimensionalparameter space �� :

��

�

��

��

�

Using this random variable, �� registration trials were considered to determinethe global transformation. The proposed algorithm (equation (13.11)) was usedto estimate the known inverse transformation and was able to recover the optimalregistration parameters in all cases. In order to have a more reliable validation,registration for the cases of occlussions as well missing components was consid-ered using the same random generator. As shown in figure (13.5) the performanceof the method was excellent for moderate and satisfactory for heavy occlussions..

13.4 Segmentation & Shape Prior Constraints

Segmentation can be the basis to many image processing and computer visionapplications [613]. Such a problem has an enormous complexity due to the varietyof conditions to be dealt with.

Propagating curves and surfaces is an attractive approach to cope with thisapplication domain. Inspired by the pioneering work of Witkin, Kass and Ter-


(i)(a)

(b)

(ii)(a)

(b)

Figure 13.5. Empirical Evaluation to Occlusions & Missing Components; (i) moderateshape deformation (one missing finger) - Registration Performance: 100 %, (ii) heavyshape deformation (two missing fingers)- Registration Performance: 77 %. (a) InitialConditions, (b) Registration Result.

zopoulos [262] numerous techniques have been proposed that aim at recoveringa curve that corresponds to the lowest potential of an objective function. Thisfunction can account for the desired image properties as well as some internalproperties of the curve.

Level set representations [401] are well suited computational methods toperform this task, tracking moving curves/interfaces. They can be used for any di-mension (curves, surfaces, hyper-surfaces, ...), are parameter free and can changenaturally the topology of the evolving interface. Moreover, they provide a naturalway to determine and estimate geometric properties of the evolving interface.

Our visual space consists of rigid as well as non-rigid objects. Performance ofgeometric flows when implemented using level set methods for non-rigid objectsis expected to be exceptional since these methods can account for very local de-formations [81, 330, 415]. On the other hand sensitivity to noise as well as toocclusions when rigid objects are considered is expected to be poor.

Medical imaging is an area of growing attention in the vision community.Many structures of interest in this domain cannot be considered neither rigid nornon-rigid. One can claim that they follow some global shape consistency, whileimportant local deformations can also be present. Recovering these deformations


is of great importance and therefore geometric flows implemented using level setsis a promissing technique. Deformable models/templates [50, 360] is an alterna-tive that can account for global shape consistency and local variations when themodel is represented using an important number of basic functions.

Introducing global shape constraints according to a flexible model while pre-serving the ability of dealing with local deformations when level set-drivenmethods are used is a challenging perspective [111, 310, 444, 526]. Towards thisend, we will combine the shape model and the registration components intro-duced in the previous sections to deal with a limited segmentation problem; theextraction of objects of particular interest.

Given an image and a model for a structure of interest, one would like to recoverthe image region that corresponds to this structure. Such region should followsome shape as well as visual-driven intensity properties. If we assume that theregion is known, then it may be helpful also to recover the transformation be-tween this region and the shape model. In medical imaging diagnostic purposescan require such information where important local or global deviations from theshape can be used as an indicator for diseases. Our approach will be based onthe propagation of curves and address both issues simultaneously. Recoveringthe image region that corresponds to the structure of interest and registering thesegmentation result to the prior shape model.

13.4.1 Prior Global Shape Constraints

Inspired by the existing geometric flows, one can assume the propagation of aninitial curve/interface� � �� using an implicit level set representation� accord-ing to some data driven terms towards the optimal segmentation result accordingto the following assumptions;

� Visual support had to be used to derive the optimal segmentation map;

� This representation is evolving while respecting the global shape propertiesof the object to be recovered. In other words the evovling contour had to beregistered to the prior model;

� Given the registration between the evovling representation and the priorshape, one should refine its local form to better account for the prior shapeknowledge while respecting the global visual properties of the object.

Then, rigid registration between the evolving representation and the priormodel can be recovered. However, one can assume that this global transforma-tion is known. Then, evolving locally the level set representation to better matchwith the prior model given the correspondence map (registration) is a task that hasto be considered

In order to facilitate the introduction of a segmentation module that can accountfor global shape consistency given a model��, one can define the approximations


of Dirac and Heaviside [607] distributions as:

Æ��

��

��

�

��

�� cos

��

�

��

��

��

��

��

�

��

�

��

�

�sin

��

�

��

Then it can be shown easily that

�� Æ��

The approximations of Dirac [Æ��] and Heaviside [��] distributions can beused to introduce optimization criteria in the space of level set representationsas shown in [607]. Towards this end, given a level set representation , one candefine boundary as well as region-driven modules;

��

��

��

� � ��

�

��

Æ��

��

��

��

�

��

��

(13.15)

where � and � are region and boundary attraction functions.In order to introduce the shape constraint, one can assume that all instances

of the evolving representation belong to the family shapes that is generated byapplying all possible global transformations (according to a predefined model)to the prior shape model. Then, there is an ideal transformation � between theprior model � and the observed representation that satisfies (similar to theregistration case) the following condition;

��

where is an evolving representation. In order to better introduce our globalshape consistency component, the absence of visual information will be consid-ered and the problem can be stated as follows; given a initial contour and its levelset representation perform a pixel-wise propagation that transforms this contourto a structure that belongs to the family of shapes defined by the shape prior. Onecan consider the same objective function as the one used for the registration;

��

��

��

�� (13.16)


with an augnebted set of unknown variables; the transformation� and the level setrepresentation �. Then, a gradient descent method with respect to the registrationparameters as well as to � can be used to recover their optimal estimates. Themotion equations with respect to the transformation � are similar to the onespresented in equation (13.9) while the level set representation � has to evolveaccording to:

�

�� Æ�� (13.17)

A detailed interpretation of this flow can be found in [444]. The second compo-nent of this flow is a constant deflation force that aims at shrinking the evolvinginterface and consequently decreasing the load of the objective function. Thiscomponent can be ignored. Some results on the performance of this flow forpreserving a global shape consistency within the propagation of level set rep-resentations along with the projection to the prior model (registration) are shownin figure (13.6).

Shape variability is not considered in this model and one can claim that the priorconstraint is hard. Given the definition of the prior model, one can easily modifythe objective function to better account for shape variations. The stochastic levelset representation �� introduced in equation (13.2) can be used and onecan seek the maximum likelihood between the evolving representation � and theprior model leading to the following condition;

��

��

��

��

(13.18)

Recovering� and� according to the Maximum Likelihood criterion is equivalentwith minimizing the [-log] function leading (after the subtraction of the constantterms) to;

��

��

��

��

��

��

��

��

��

(13.19)

The interpretation of the above function is different to the one used for registra-tion. The shape is considered in a qualitative manner. Shape components with lowvariance correspond to areas where the training examples are perfectly alignedduring the learning phase. It is natural that these components are more importantand therefore they tend to load the objective function more than the ones withhigh variability. As a consequence, registration as well as local propagation it tobe done efficiently for the shape parts with low variability. A gradient descent


method can be used to recover the registration parameters3 and the flow to beused for the deformation of the evolving representation � (the deflation force isnot considered).

�

��

��

��

�

��

��

��

��

��

��

��

��

� (13.20)

where the image coordinates �� have been ommited from to the registrationflow. The introduction of the shape uncertainties to the segmentation/registrationprocess can further improve the registration performance especially when globaltransformations are considered for objects with non-rigid components.

Robust estimators have been used to deal with the non-rigid components whenglobal registration is considered. The shape variability map provides a naturalway to better account for their inconsistency with the global motion. Accordingto these (robust) estimators, the condition that at least 50% of the samples supportsthe dominant case has to be met in order to recover reliable registration estimateswhen outliers are present. The use of variability maps (as it will be shown later,see equation (13.19)) can remove this constraint and theoretically can provideaccurate estimates with significantly less data support.

13.4.2 Visual Support, Shape Constraints & Segmentation

Numerous variational frameworks based on the propagation of curves using levelset representations have been proposed for image segmentation [11, 84, 100,270, 456, 593] . Our prior shape model can be naturally integrated with theseframeworks as an additional shape-driven term along with data-driven terms.

Following our previous work, we will consider this integration within thegeodesic active region model [404, 409, 411] that aims at combining boundary(in the form of Geodesic Active Contours [84, 270]) with some regional/globalproperties of the object to be recovered. The original model was defined on theimage plane, and the obtained motion equations were implemented using levelset representations. Here, we will introduce a shape-constrained version of thismodel directly on the level set representation space. We will consider the bi-modalsegmentation case to facilitate the introduction.

An evolving level set representation can be used to define an image partitioninto two regions, the inner region�� and the background�� . Giventhis partition, one can estimate some global intensity (region-based) descriptors

3The motion equation of scale component is presented. Same derivations are applicable for theother unknown variables of the rigid transformation.


(i) (ii) (iii) (iv)

Figure 13.6. Knowledge-based Image Segmentation (raster-scan format; (i, iii) contourpropagation in the image plane presented in a raster0scan format, (ii, iv) registration of thecurrent result to the prior model using the projection of the evolving interface to the priormodel according to the current estimates of the global rigid model parameters.

�� that can account for the desired visual/intensity properties (underlying dis-tributions) of these regions. Such descriptors can capture the visual properties ofthe structure of interest and can assumed to be known or determined and updatedwithin the process. The geodesic active region model aims at partioning the imageaccording to boundary as well region-driven criteria;

��

��

Æ��

��

��

��

��

��

��

��

��

��

��

(13.21)

where �� are constants balancing the contribution of the different terms.The first two terms account for visual consistency properties4 of the segmentationmap (region of interest, background) [613], the third term stands for the bound-ary attraction (�� is a monotonically positive decreasing function that can bereplaced with more efficient edge detectors [151]) [84, 270] and the last term isthe shape prior applied to the interface [444]. As shown in [444], the method canbe extended to deal with � objects using the same number of evolving functionsunder the assumption that � shape prior models are available.

A detailed interpretation of this objective function can be found in [409, 444].Conceptually, we are seeking for a curve (level set representation) of minimallength attracted by the region boundaries, that defines an image partition sup-

4Homogeneity is a particular case of this condition.


Figure 13.7. Shape Priors, Level Set Representations for the real-time segmentation ofultrasonic images (temporal results presented in a raster-scan format.

ported by the visual/intensity information given some prior knowledge of thedesired visual properties of the structure of interest and follows global shapeconsistency according to a known prior model.

13.5 Discussion

In this chapter we have explored some aspects of shape analysis. Level setrepresentations have been used for shape representation and have been inte-grated within a variational framework to deal with registration. A novel termfor introducing global shape constraints was proposed to better account for thesegmentation problem when occlusions or corrupted information is to be dealtwith.

The use of non-parametric level set models is required for family of shapes withsignificant variability. This issue is to be addressed along with the integration ofintensity properties to an improved version of shape model that does account forthe geometric as well as visual properties of the object to be recovered. Alternativetechniques like the use of mutual information are currently under investigation toperform real-time segmentation/registration and tracking. Changes of topology isthe most solid argument for using level set methods. Our approach to image seg-mentation doesn’t support changes of topology within the prior shape model. Onecan claim that this is an important limitation of the method. The implicit repre-sentation of curve using distance functions is a powerful way to describe shapesand provides a natural way to estimate their geometric properties. Such proper-ties make these techniques suitable to a large number of applications. A naturalextension is to be able to deal with multiple objects using a single level set repre-


sentation that can change the topology while respecting some shape properties ofthe objects to be recovered.

Acknowledgments

The author would like to acknowledge Benedicte Bascle, Marie-Pierre Jolly,Visvanathan Ramesh and Chenyang Xu for fruitful discsussion and suggestions.

Documents

Variational Principles in Optical Flow Estimation and Tracking