Matching of medical images by self-organizing neural networks

  • Published on

  • View

  • Download

Embed Size (px)


  • aetw

    Giuseppe Coppini a,*, Stefano Diciotti b, Guido Valli b

    noise indicate considerable robustness against statistical variability. Applications to clinical images are presented.

    preattentive processes to attentive tasks. A wide of paramount importance in modern medical

    imaging. In the latter case, it is necessary to buildcomprehensive functional and anatomical repre-

    sentations of the observed biological structures by

    utilizing the partial views oered by basic imaging

    procedures such as MRI, PET, CT (van den Elsen

    qPartially founded by the Italian Ministry for Education,

    University and Research (MIUR).*

    Pattern Recognition Letters 25Corresponding author. Tel.: +39-503-153-480; fax: +39-

    503-152-166. 2003 Elsevier B.V. All rights reserved.

    Keywords: Image matching; Medical imaging; SOM neural network; Gabor wavelets

    1. Introduction

    Finding correspondences between images is a

    central problem in many vision activities, from

    category of image matching problems arises from

    the need to integrate images generated by dierent

    modalities. The importance of integration tasks

    has rapidly grown in recent years and has becomea Consiglio Nazionale della Ricerche (CNR), Institute of Clinical Physiology, Via Moruzzi 1, 56124 Pisa, Italyb Department of Electronics and Telecommunications, University of Florence, Italy

    Received 11 March 2003; received in revised form 21 October 2003


    A general approach to the problem of image matching which exploits a multi-scale representation of local image

    structure and the principles of self-organizing neural networks is introduced. The problem considered is relevant in

    many imaging applications and has been largely investigated in medical imagery, especially as regards the integration of

    dierent imaging procedures.

    A given pair of images to be matched, named target and stimulus respectively, are represented by Gabor Wavelets.

    Correspondence is computed by exploiting the learning procedure of a neural network derived from Kohonens SOM.The SOM units coincide with the pixels of the target image and their weight are pointers to those of the stimulus images.

    The standard SOM rule is modied so as to account for image features. The properties of our method are tested by

    experiments performed on synthetic images. The considered implementation has shown that is able to recover a wide

    range of transformations including global ane transformations and local distortions. Tests in the presence of additiveMatching of medical imneural nE-mail addresses: (G. Coppini), diciotti@ (S. Diciotti), (G. Valli).

    0167-8655/$ - see front matter 2003 Elsevier B.V. All rights reservdoi:10.1016/j.patrec.2003.10.012ges by self-organizingorks q

    (2004) 341352 al., 1993). Several correspondence problems

    have been faced which include mono-modal or


  • subject variability, nevertheless the observed

    structures keep some general properties such asconnectedness and proximity among sub-struc-

    tures. In general, it seems reasonable to search for

    a correspondence law which is unique from T toS, 1 and which maps features in the T plane tosimilar features in the S plane by matching spa-tially contiguous features in T into spatially con-tiguous features in S. These conditions can berestated more concisely by saying that the desiredmapping is a feature- and topology-preserving

    transformation from T into S. The importance oftopology preservation in image matching has been

    ognition Letters 25 (2004) 341352multi-modal image correspondence and intra- or

    inter-subjects matching. In addition, matching of

    3D images has become an important issue along

    with the mapping of imaged structures to reference

    representations such as anatomical atlases. Much

    eort has been devoted to nding global defor-mations, which has led to ecient algorithms

    being able to recover linear transformations. A

    popular method is based on the maximization of

    the mutual information of the two images (Wells

    III et al., 1996). More recently, the use of deform-

    able models has been widely investigated so as to

    recover local non-linear deformations. A number

    of references on the subject can be found in(Maintz and Viergever, 1998). Thermodynamical

    analogy with Maxwell demons has been also

    investigated (Thirion, 1998).

    In general, specic solutions have been pro-

    posed for many matching problems. The related

    algorithms usually exploit some form of prior

    knowledge, and are often based on strong

    assumptions. On the other hand many corre-spondence nding problems share important

    common facets, such as the statement of image-

    similarity criteria or the denition of adequate

    computational schemes. In this view, the impor-

    tance of general solutions for image matching

    problems has been considered by several

    researchers (Haralick and Shapiro, 1993). In par-

    ticular, we believe that a self-organizing process isa natural and very promising setting. As reported

    by Bellando and Kotari (1996), the computation of

    topology preserving maps using Kohonen neural

    networks (Kohonen, 2001) can provide valid

    solutions to establishing image correspondence.

    Wurtz et al. (1999) compare the behavior of Ko-

    honens SOM with the Dynamic Link Architecture(Konen et al., 1994) paying particular attentionto the computational eciency of self-organizing

    processes. It is worth mentioning that the appli-

    cation of a SOM network to image registration is

    described by Huwer et al. (1996).

    In this work we will take the following general

    problem into account. Let Itr and Isr0, twoimages (target and stimulus image, respectively)

    with r x; y, and r0 x0; y0 co-ordinate vectorsdened in proper subsets T and S, respectively

    342 G. Coppini et al. / Pattern Rec(image planes) of R2 space. We assume that afeature vector ftr ff it rg describing relevantproperties of Itr can be computed for each pointr in T . Similarly, fsr0 ff is r0g will indicate thefeature vector of Isr0 for each r0 2 S. We searchfor a correspondence rule M : T 7!S that mapspoints in T to points in S which have similarmorphological properties, as described by the

    feature vectors (see Fig. 1). This is an ill-posed

    problem, and further constraints must be consid-

    ered to compute a useful solution, when stated so.We believe that powerful constraints can be de-

    rived from the regularity of the observed world

    even if the use of specic knowledge available for

    each matching problem considered can lead to

    adequate solutions. For example, images of nor-

    mal anatomical structures exhibit a large inter-

    tI (r)

    sI (r)




    Sstimulus image

    target image

    Fig. 1. A general matching procedure is expected to preserve

    pictorial features and their neighborhood relationship.1 One-to-one mapping can be desirable in some cases.

  • given in the image planes I fT ; Sg and the fea-

    ble image description to compute a useful set of

    pictorial features f ff ig. This led us to investi-gate a self-organizing neural network coupled with

    ognition Letters 25 (2004) 341352 343ture space F , respectively. Let us assume that forany given r1; r2 2 T , r01; r02 2 S are the correspond-ing transformed points, i.e. r01 Mr1, r02 Mr2. We will say that the transformation Mpreserves image topology if the following condi-

    tions are met:

    (1) Preservation of image plane topology:

    limdI r1;r2!0

    dIr01; r02 0

    (2) Feature preservation:

    limdI r1;r01!0

    dF ftr1; fsr01 0

    Though further constraints can be taken into

    account (such as the high order smoothness of

    the mapping), our aim is the investigation of the

    preservation of the neighborhood relationship.

    To this end we consider the computation of

    image matching by using a topology preserving

    neural network derived from Kohonens SOM.The resulting computation is typically data-drivenand no specic prior model is expected to con-

    strain the obtained solution. In this sense our

    approach is typically non-parametric. In order to

    investigate the related capabilities, we outline the

    basic scheme of the adopted computational para-

    digm in the next section. Subsequently we present

    and analyze the performances observed when a

    set of known mathematical transformations wasapplied to phantom images in the presence of

    additive noise. Applications to computing corre-

    spondence between biomedical images are also


    2. Computational paradigm

    Following the previous considerations, a gen-

    eral framework of image matching problems re-

    quires: (a) a proper computational architecture todiscussed by Musse et al. (2001) who propose the

    use of hierarchical parametric models. A formal

    denition of the meaning of topology preserving

    transformation is useful for the following. To this

    end we assume that proper metrics dI and dF are

    G. Coppini et al. / Pattern Recimplement a topology preserving map, (b) a exi-image features computed by the Gabor Wavelet


    2.1. Matching through self-organization

    The computation of topology preserving maps

    between vector spaces can be carried out by using

    Kohonen neural networks (Kohonen, 2001).

    Basically, they include a grid of units, typically 2D,

    each of which receives the same input vector

    (stimulus) and operates on a winner-takes-allbasis. For a given input, the network weights are

    changed according to the rule: (i) nd a winner

    unit, and (ii) change the weights of the units in a

    neighborhood of the winner unit, the size of the

    neighborhood (dening the extent of lateral plas-

    ticity) is a decreasing function of time. The relax-

    ation process obtained by iteratively applying the

    above rule for a sequence of input stimuli is thebasis of the training of ordinary Kohonen net-

    works. In this work, the same process, properly

    adapted, is exploited to compute the desired cor-

    respondenceM. In our computational scheme, thepixels of one of the two images Itr (called thetarget image) correspond to the units of the Ko-

    honen network. In general we assume the image to

    be sub-sampled by using a N N grid of pixels,which provides a N N SOM. The weights of eachSOM unit can be interpreted as pointers to the

    second image Isr0 (called stimulus image) whosecoordinates are the input of the network (Fig. 2).

    In this way, the weight vectors directly give the

    displacement needed to match point r with the cor-

    responding point r0. For this reason, in the fol-lowing the network weight of the ith unit 2 isindicated byMri, while r0 is the input (or stimu-lus) vector. The standard Kohonens algorithmthat governs the updating of network weights in

    response to a stimulus r0 presented at time t is asfollows:

    2 Without loss of generality we will assume that each unit isidentied by a predened index.

  • ognitii(i) locate the winner unit c by:

    c argminkkr0 Mrkk

    (ii) change the network weights according to theequation:

    Mt1i ri Mtiri athictr0 Mtiri

    The term at is the learning rate which is oftengiven by:

    at a0 tmax ttmaxbeing a06 1 and tmax the (prexed) number ofiterations of the relaxation process. The function

    Fig. 2. Image matching by a 2D self-organizing network: SOM

    units are superimposed to the pixels of the target image and the

    network is made to relax under the action of the pixel of the

    stimulus image.

    344 G. Coppini et al. / Pattern Rechic accounts for lateral plasticity: its value is onefor i c and falls o with distance kri rck fromthe winner unit. Gaussian is a widely used neigh-

    borhood function:

    hict exp" kri rck



    The parameter rt determines the width of theneighborhood and controls the topology preser-

    vation process. As in the case of the learning rate,

    rt is decreased as the relaxation progresses.Unfortunately, the previous self-adaptation

    equation is too simple to produce the desired re-

    sults. Only condition 1 of our denition (i.e.

    preservation of image plane topology) is satisedby the obtained mapping. On the other hand,

    image features are not taken into account and aunitary mapping is produced independently of the

    image content. In order to achieve a meaningful

    behavior, i.e. to ensure that feature preservation is

    satised (condition 2), we modied the standard

    SOM rule so as to include image features. To be

    more precise, we dened feature similarity S by:

    Sfsr0; ftr 11 kfsr0 ftrk

    where k k is the ordinary Euclidean metric. ThefunctionS assumes a unit value when fsr0 ftrand drops to zero as kfsr0 ftrk increases.Having dened S, we replaced the SOM weightupdate rule with the equation:

    Mt1i ri Mtiri athict

    r0 MtiriSfsr0; f tr 1

    With this choice, maximal weight changes are al-

    lowed when similarity between corresponding

    features is high. On the other hand, weight changes

    are weakened when dierent image features arecoupled. Otherwise stated, image features inu-

    ence the computed map by simply modulating the

    entity of weight update. One can have an idea of

    the eect of such a process by considering the ex-

    treme case of zero-valued similarity: this prevents

    any weight change despite the competition mech-

    anism that has no actual eect. In brief, competi-

    tion among network units involves spatialcoordinates only, while changes to mapping are

    controlled by similarity of image-feature.

    2.2. Image representation by Gabor Wavelets

    In general, the matching process can be based

    on pictorial features which are optimized with re-

    spect to the problem to be faced. Nevertheless,especially in the absence of explicit knowledge,

    several low-level descriptions of images are known

    (such as derivative of Gaussians, and several

    classes of wavelets) which are well suited to cap-

    ture image structures. In this view, we took Gabor

    Wavelets into account. They provide a multi-scale

    image representation which can be rigorously

    framed in the theory of wavelet transform (Lee,1996) while keeping an intriguing biological

    inspiration (Daugman, 1988). Gabor Wavelets for

    on Letters 25 (2004) 341352an image Ir can be obtained by a set of lters:

  • Ir Wr; kwith:

    Wr; k 4k2 exp2k2r2

    expi2pk r

    exp p



    where k kx; ky is the spatial frequency vectorwhich is usually more conveniently expressed in a

    polar reference k k;u where the modulus kacts as a scale-selection parameter. The gaborian

    kernel, for a xed k vector, is a Gaussian which is

    modulated by a complex sinusoid. Such a kernelcan be decomposed into a pair of quadrature l-

    dierent scales, image textures with dierent

    coarseness are enhanced. In addition, Gabor ker-

    nels are well suited to model the receptive elds of

    simple cells in biological vision systems. Moreover,

    the modulus mr; k of the output of a pair ofGabor lters:

    mr; k;u mr; k

    cr; k Ir2 sr; k Ir2

    qis usually considered a good approximation of the

    output of complex cells (Spitzer and Hochstein,

    1985). The non-linearly ltered images obtained by

    mr; k jIr Wr; kj are used in this work.

    G. Coppini et al. / Pattern Recognition Letters 25 (2004) 341352 345ters:

    cx; y; k;u 4k2 exp2k2x2 y2

    cos2pkx cosu y sinu expp2=2

    sx; y; k;u 4k2 exp2k2x2 y2

    sin2pkx cosu y sinu

    These are two Gaussians with a standard deviation12k, each Gaussian being modulated by a sinusoid

    with a frequ...