Download pdf - VISIONTRAIN David Gustavsson Spatial and Temporal Inpainting · MRTN-CT-2004-005439 PhD midterm report D. Gustavsson 4 Scientiﬁc achievements and Results 4.1 FRAME for Inpainting

SIXTH FRAMEWORK PROGRAMME

MARIE CURIE ACTIONS

HUMAN RESOURCES ANDMOBILITY

MARIE CURIE RESEARCH TRAINING NETWORKS (RTN)

COMPUTATIONAL AND COGNITIVE V ISION SYSTEMS:A TRAINING EUROPEANNETWORK

VISIONTRAIN

David Gustavsson

Spatial and Temporal Inpainting

Contract number:MRTN-CT-2004-005439Project start/end dates:1/5/2005 – 30/4/2009

Reporting period:1/5/2005 – 30/4/2007

Phd advisor:Mads Nielsen([email protected])Host institution: ITU/DIKU (partner 4/12), Denmark

MRTN-CT-2004-005439 PhD midterm report D. Gustavsson

Contents

1 Introduction: Inpainting 2

2 Brief review of the state of the art 2

3 Approach and methodology 2

4 Scientific achievements and Results 34.1 FRAME for Inpainting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . 34.2 Occlusion Detection by Contour Flows . . . . . . . . . . . . . . . .. . . . . . . . . 3

5 Future work 6

6 List of publications 6

A Full text of published and accepted papers 7

1


1 Introduction: Inpainting

Image inpainting concerns the problem of reconstruction ofthe image contents inside a regionΩwith unknown or damaged contents. The reconstruction is based on the available surrounding imagecontents. An important difference between inpainting and denoising is: In inpainting all the imagecontent in the missing region is unknown, in denoising a noisy version of the image is given and thenoise is often assumed to be additive.

Other applications of inpainting are : object removal, super-resolution and deinterlacing.

2 Brief review of the state of the art

Common inpainting methods are (broadly categorised):Functional minimisation methods: Inpainting methods based on minimising some smoothness

criteria. An energy functionalE(u) is constructed such that the functionu that minimise the func-tional provides the inpainting. Common smoothness criteria are the Total Variation| ∇u | and Har-monic | ∇u |2, but many other criteria with different properties can be used. Calculus of variationsand the Euler-Lagrange equation are often used to find theu that minimise the energy functional.Functional minimisation methods reconstruct geometric structure quite well, but have an tendencyto produce a too smooth solution in textured regions. ([3, 4]contain overviews of this category ofmethods.)

Synthetic texture reproduction methods: Methods based on reproduction of texture from asample and by a statistical texture model. The regionΩ is reconstructed by reproducing the textureaccording to the texture model. Texture based methods are good at reconstructing texture but usuallyfails to reconstruct large scale geometric structures. ([2, 7, 8, 12, 13] are example of texture basedmethods.)

Exemplar-Based methods: Patches from the non-damage region are pasted into the missingregionΩ based on a pasting order and a similarity measure [5, 6].

3 Approach and methodology

The two main objectives for the PhD-project are:Objective OneDevelop a general spatial functional based method that could inpaint regions con-

taining textures.Objective Two Extend the method to the temporal dimension in such a way thatit could inpaint

geometric structures and textures in sequences of images.

2


4 Scientific achievements and Results

4.1 FRAME for Inpainting

FRAME (Filters, Random fields, And Maximum Entropy) is a well-known method for analysing andreproducing textures [12, 13]. FRAME constructs a probability distributionp(I) for a texture fromobserved sample images.

Given a set of filtersF α(I) one computes the histogram of the filter responsesHα with respect tothe filterα. The filter histograms are estimates of marginal distributions of the full probability distri-butionp(I). Given the marginal distributions for the sample images onewants to find all distributionsthat have the same expected marginal distributions, and among those find the distribution with max-imum entropy, i.e. by applying the maximum entropy principle. The hereby constructed probabilitydistribution is a Gibbs distribution.

Gustavsson et al. [10, 11] contribution is to introduce a temperature termT into the Gibbs distri-bution learnt by FRAME. By sampling from the distribution using different temperatures, differentimage contents are added. By sampling using a smallT value - a cooled distribution - fine scaletextures are suppressed, while coarse scale structures arereconstructed. By sampling from the distri-bution using a high temperatureT - a heated distribution - fine scale textures are added.

A two step method for inpainting is proposed :

1. Cooling Phase:By letting the temperatureT → 0 (simulated annealing) a maximum a poste-riori (MAP) like solution is found. The MAP-like solution reconstruct coarse scale geometricstructure of the image, while it suppress the fine scale textures.

2. Heating Phase: By letting the temperatureT → 1 fine scale textures will be reconstructedwithout destroying the coarse scale geometric structure.

The cooling phase reconstruct the geometric structure, while the heating phase reconstruct thetexture. Both phases are necessary for a visual appealing reconstruction. Examples of inpaintingresults for the different phases can be found in figure 1.

4.2 Occlusion Detection by Contour Flows

Shape information in the form of shape priors is an importanttool for object segmentation in im-age/video sequences. Fundana et al. [9] show that by using the previous segmentation as a shapeprior, non-rigid objects can be segmented even when large parts of the individual objects are oc-cluded. By using this approach the contour of an object can befound accurately even in the occludedpart of the object. In figure 2, the segmentation of an occluded non-rigid moving object using theprevious segmentation as shape prior is shown.

The motion and deformation of an object can be estimated by computing a displacement fieldthat maps the initial object contourC1 onto the object contourC2 in the following frame. By usingregistration by geometry-constrained diffusion [1] a displacement field, including both motion anddeformation, is computed for the whole image.

3


Figure 1: Inpainting using FRAME and the two step Cooling/Heating Scheme. From left to right a)the image with the unknown regionΩ, b) after the first phase: the geometric structure is reconstructedby a MAP-like solution, c) after the second phase: the finer scale texture is reconstructed by heatingthe distribution, and d) the reconstruction in context.

A novel variational motivation for the geometry-constrained diffusion is presented. The displace-ment field is constrained by the property that contourC1 is mapped ontoC2. This allows pixels insidecontourC1 to be mapped to pixels inside contourC2. The displacement field of the contour can beviewed as an estimation of the motion and deformation of the object between the frames and for pixelsinsideC1 the displacement is an estimation of how the intensities aredisplaced between the frames.

The pixels (insideC2) in the following frame can be predicted by deforming the first frame ac-cording to the displacement field. If the predicted intensities are similar to the intensities found inthe next frame then there is no occlusion. If the difference between the predicted and the observedintensities are large then an occlusion is detected. (Submitted to ACCV-07.)

Sketch of the Occlusion Detection Algorithm:

1. Displacement Field:Compute the displacement field between two frames given the contour ofthe object using Geometry-Constrained diffusion.

2. Deform the frame: Deform the first frame using the displacement field.

4


Figure 2: Segmentation of a non-rigid moving object in a sequence using the previous segmentationas a shape prior. By using shape priors in the segmentation process the object can be segmented evenwhen a large part of the region is occluded.

3. Compare the intensities:Compare the intensities between the deformed frame and the follow-ing frame for pixels inside contourC2. Large differences imply that the object is occluded.

Results from an artificial sequence are presented in figures 2and 3, results from a sequence con-taining a walking person are presented in figure 4.

The research was done together with Ketut Fundana (VISIONTRAIN PhD-student), Niels-ChristianOvergaard and prof. Anders Heyden (VISIONTRAIN member), during a 3 month guest visit atMalmö University, visiting prof. Anders Heyden

Figure 3: Occlusion detection inside the segmented object by comparing the observed intensities withthe predicted intensities. The intensities inside the second contour can be predicted by computing thedisplacement (motion and deformation) between two frames.

5


Figure 4: Occlusion detection inside the contour of the walking person by comparing the observedand the predicted intensities. The intensities inside the second contour can be predicted by computingthe displacement (motion and deformation) between two frames.

5 Future work

During the next period will the relation between the statistical texture based methods and the energyfunctional minimisation methods be investigated. Especially, the relation between the MAP-solutionsand the image contents in the MAP reconstruction will be investigated.

I will also continue working on inpainting in the temporal domain. Especially the focus will beon bringing together optical flow with the results from the visit in Malmö, and how this can be usedfor temporal inpainting. It will also be investigated how statistical models can be used for inpaintingin the temporal domain.

During the next period I will visit prof. Christoph Schnörr at University of Mannheim.

6 List of publications

Published papers:

• David Gustavsson, Kim S. Pedersen, and Mads Nielsen. Geometric and texture inpainting bygibbs sampling. InProceedings of Swedish Symposium in Image Analysis 2007, 2007.

• David Gustavsson, Kim S. Pedersen, and Mads Nielsen. Image inpainting by cooling andheating. InProceedings of Scandinavian Conference on Image Analysis 2007, 2007.

References

[1] Per Andresen and Mads Nielsen. Non-rigid registration by geometry-constrained diffusion. InMICCAI ’99: Proceedings of the Second International Conference on Medical Image Comput-ing and Computer-Assisted Intervention, pages 533–543, London, UK, 1999. Springer-Verlag.

[2] J. S. De Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images.In Computer Graphics, pages 361–368. ACM SIGGRAPH, 1997.

6


[3] Tony F. Chan and Jianhong Shen. Variational image inpainting. Communications on Pure andApplied Mathematics, 58, February 2005.

[4] Tony F Chan and Jianhong Shen.Image Processing and Analysis - variational, PDE, wavelet,and stochastic methods. SIAM, 2006.

[5] A. Criminisi, P. Pérez, and K. Toyama. Object removal by exemplar-based inpainting. InCVPR2003, volume 2, pages 721–728, June 2003.

[6] A. Criminisi, P. Pérez, and K. Toyama. Region filling and object removal by exemplar-basedimage inpainting.IEEE Transactions On Image Processing, 13(9):1200–1212, September 2004.

[7] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. InProceedingsof SIGGRAPH ’01, Los Angeles, California, USA, August 2001.

[8] Alexei A. Efros and Thomas K. Leung. Texture synthesis bynon-parametric sampling. InIEEEInternational Conference on Computer Vision, pages 1033–1038, Corfu, Greece, September1999.

[9] K. Fundana, N.C. Overgaard, and A. Heyden. Variational segmentation of image sequencesusing deformable shape priors. InSCIA-2007, 2007.

[10] David Gustavsson, Kim S. Pedersen, and Mads Nielsen. Geometric and texture inpainting bygibbs sampling. InSSBA-2007, 2007.

[11] David Gustavsson, Kim S. Pedersen, and Mads Nielsen. Image inpainting by cooling and heat-ing. In SCIA-2007, 2007.

[12] Song Chun Zhu, Ying Nian Wu, and David Mumford. Minimax entropy principle and its appli-cation to texture modelling.Neural Computation, 9(8):1627–1660, 1997.

[13] Song Chun Zhu, Ying Nian Wu, and David Mumford. Filters,random fields and maximumentropy (frame): To a unified theory for texture modeling.International Journal of ComputerVision, 27(2):107–126, 1998.

A Full text of published and accepted papers

7

Image Inpainting by Cooling and Heating

David Gustavsson1, Kim S. Pedersen2, and Mads Nielsen2

1 IT University of CopenhagenRued Langgaards Vej 7, DK-2300 Copenhagen S,Denmark

[email protected] DIKU, University of Copenhagen

Universitetsparken 1, DK-2100 Copenhagen Ø, Denmarkkimstp,[email protected]

Abstract. We discuss a method suitable for inpainting both large scalegeometric structures and stochastic texture components. We use the well-known FRAME model for inpainting. We introduce a temperature termin the learnt FRAME Gibbs distribution. By using a fast cooling schemea MAP-like solution is found that can reconstruct the geometric struc-ture. In a second step a heating scheme is used that reconstruct thestochastic texture. Both steps in the reconstruction process are neces-sary, and contribute in two very different ways to the appearance of thereconstruction.Keywords: Inpainting, FRAME, ICM, MAP, Simulated Annealing

1 Introduction

Image inpainting concerns the problem of reconstruction of the image contentsinside a region Ω with unknown or damaged contents. We assume that Ω is asubset of the image domain D ⊆ IR2, Ω ⊂ D and we will for this paper assumethat D form a discrete lattice. The reconstruction is based on the available sur-rounding image content. Some algorithms have reported excellent performancefor pure geometric structures (see e.g. [1] for a review of such methods), whileothers have reported excellent performance for pure textures (e.g. [2–4]), butonly few methods [5] achieve good results on both types of structures.

The variational approaches have been shown to be very successful for geo-metric structures but have a tendency to produce a too smooth solution withoutfine scale texture (See [1] for a review). Bertalmio et al [5] propose a combinedmethod in which the image is decomposed into a structure part and a texturepart, and different methods are used for filling the different parts. The struc-ture part is reconstructed using a variational method and the texture part isreconstructed by image patch pasting.

Synthesis of a texture and inpainting of a texture seem to be, more or less,identical problems, however they are not. In [6] we propose a two step methodfor inpainting based on Zhu, Wu and Mumford’s stochastic FRAME model (Fil-ters, Random fields and Maximum Entropy) [7, 8]. Using FRAME naively forinpainting does not produce good results and more sophisticated strategies are

needed and in [6] we propose such a strategy. By adding a temperature term T

to the learnt Gibbs distribution and sampling from it using two different temper-atures, both the geometric and the texture component can be reconstructed. Ina first step, the geometric structure is reconstructed by sampling using a cooled- i.e. using a small fixed T - distribution. In a second step, the stochastic texturecomponent is added by sampling from a heated - i.e. using a large fixed T -distribution.

Ideally we want to use the MAP solution of the FRAME model to recon-struct geometric structure of the damaged region Ω. In [6] we use a fixed lowtemperature to find a MAP-Like solution in order to reconstruct the geometricstructure. To find the exact MAP-solution one must use the time consuming sim-ulated annealing approach such as described by Geman and Geman [9]. Howeverto reconstruct the missing contents of the region Ω, the true MAP solution maynot be needed. Instead a solution which is close to the MAP solution may providevisually good enough results. In this paper we propose a fast cooling scheme thatreconstruct the geometric structure and approaches the MAP solution. Anotherapproach is to use the solution produced by the Iterated Conditional Modes(ICM) algorithm (see e.g. [10]) for reconstruction of the geometric structure.Finding the ICM solution is much faster than our fast cooling scheme, howeverit often fails to reconstruct the geometric structure. This is among other thingscaused by the ICM solutions strong dependence on the initialisation of the al-gorithm. We compare experimentally the fast cooling solution with the ICMsolution.

To reconstruct the stochastic texture component the Gibbs distribution isheated. By heating the Gibbs distribution more stochastic texture structureswill be reconstructed without destroying the geometric structure that was re-constructed in the cooling step. In [6] we use a fixed temperature to find asolution including the texture component. Here we introduce a gradual heatingscheme.

The paper has the following structure. In section 2 FRAME is reviewed, insection 2.1 filter selection is discussed and in section 2.2 we explain how FRAMEis used for reconstruction. Inpainting using FRAME is treated in section 3. Insection 3.1 a temperature term is added to the Gibbs distribution, the ICMsolution and fast cooling solution is discussed in sections 3.2 and 3.3. Addingthe texture component by heating the distribution is discussed in section 3.4.In section 4 experimental results are presented and in section 5 conclusion aredrawn and future work is discussed.

2 Review of FRAME

FRAME is a well known method for analysing and reproducing textures [8, 7].FRAME can also be thought of as a general image model under the assump-tions that the image distribution is stationary. FRAME constructs a probabilitydistribution p(I) for a texture from observed sample images.

Given a set of filters Fα(I) one computes the histogram of the filter responsesHα with respect to the filter α. The filter histograms are estimates of marginaldistributions of the full probability distribution p(I). Given the marginal dis-tributions for the sample images one wants to find all distributions that havethe same expected marginal distributions, and among those find the distributionwith maximum entropy, i.e. by applying the maximum entropy principle. Thisdistribution is the least committed distribution fulfilling the constraints givenby the marginal distributions. This is a constrained optimisation problem thatcan be solved using Lagrange multipliers. The solution is

p(I) =1

Z(Λ)exp−

∑

i

∑

α

λαi Hα

i (1)

Here i is the number of histogram bins in Hα for the filter α and Λ = λαi are

the Lagrange multipliers which gives information on how the different values forthe filter α should be distributed. The relation between λα:s for different filtersFα gives information on how the filters are weighted relative to each other.

An Algorithm for finding the distribution and Λ can be found in [7]. FRAMEis a generative model and given the distribution p(I) for a texture it can be usedfor inference (analysis) and synthesis.

2.1 The Choice of Filter Bank

We have used three types of filters in our experiments: The delta filter, thepower of Gabor filters and Scale Space derivative filters. The delta, Scale Spacederivative and Gabor filters are linear filters, hence Fα(I) = I ∗ Fα, where ∗denotes convolution. The power of the Gabor filter is the squared magnitudeapplied to the linear Gabor filter.

The Filters Fα are:

– Delta filter - given by the Dirac delta δ(x) which simply returns the intensityat the filter position.

– the power of Gabor filters - defined by | I ∗Gσe−iωx |2, where i2 = −1. Herewe use 8 orientations, ω = 0, π

4 , π2 , 3π

4 , π, 5π4 , 3π

2 , 7π4 and 2 scales σ = 1, 4, in

total 16 Gabor filters have been used.– Scale space derivatives - using 3 scales σ = 0.1, 1, 3 and 6 derivatives Gσ, ∂Gσ

∂x,

∂Gσ

∂y, ∂2Gσ

∂x2 , ∂2Gσ

∂y2 , ∂2Gσ

∂x∂y.

For both the Gabor and scale space derivative filters the Gaussian aperturefunction Gσ with standard deviation σ defining the spatial scale is used,

Gσ(x, y) =1

2πσ2exp

(

−x2 + y2

2σ2

)

.

Which and how many filters should be used have a large influence on thetype of image that can be modelled. The filters must catch the important visualappearance of the image at different scales. The support of the filters determines

a Markov neighbourhood. Small filters add fine scale properties of the image,while large filters add coarse scale properties of the image. Hence to modelproperties at different scales, different filter sizes must be used. The drawbackof using large filters is that the computation time increases with the filter size.On the other hand large filters must be used to catch coarse scale dependenciesin the image.

Gabor filters are orientation sensitive and have been used for analysing tex-tures in a number of papers and are in general suitable for textures (e.g. [11,12]). By carefully selecting the orientation ω and the scale σ, structures withdifferent orientations and scales will be captured.

It is well known from scale space theory that scale space derivative filterscapture structures at different scales. By increasing σ in the Gaussian kernel,finer details are suppressed, while coarse structures are enhanced. By using thefull scale-space both fine and coarse scale structures will be captured [13].

2.2 Sampling

Once the distribution p(I) is learnt, it is possible to use a Gibbs sampler tosynthesise images from p(I). I is initialised randomly (or in some other waybased on prior knowledge). Then a site (x, y)i ∈ D is randomly picked andthe intensity Ii = I((x, y)i) at (x, y)i is updated according to the conditionaldistribution [14, 10]

p(Ii|I−i) (2)

where the notation I−i denotes the set of intensities at the set of sites (x, y)−i =D\(x, y)i. Hence p(Ii|I−i) is the probability for the different intensities in site(x, y)i given the intensities in the rest of the image. Because of the equivalencebetween Gibbs distributions and Markov Random Fields given a neighbourhoodsystem N (the Hammersley-Clifford theorem, see e.g. [10]), we can make thesimplification

p(Ii|I−i) = p(Ii|INi) (3)

where Ni ⊂ D\(x, y)i is the neighbourhood of (x, y)i. In the FRAME model,the neighbourhood system N is defined by the extend of the filters Fα.

By sampling from the conditional distribution in (3), I will be a sample fromthe distribution p(I).

3 Using FRAME for inpainting

We can use FRAME for inpainting by first constructing a model p(I) of theimage, e.g. by learning from the non-damaged part of the image, D\Ω. We thenuse the learnt model p(I) to sample new content inside the damaged region Ω.This is done by only updating sites in Ω. A site (x, y)i ∈ Ω is randomly pickedand updated by sampling from the conditional distribution given in (3). If thesite (x, y)i is close (in terms of filter size) to the boundary ∂Ω of the damagedregion, then the filters get support from both sites inside and outside Ω. The sites

outside Ω are known and fixed, and are boundary conditions for the inpainting.We therefore include a small band region around Ω in the computation of thehistograms Hα. Another option would have been to use the whole image I tocompute the histogram Hα, however this has the downside that the effect ofupdates inside Ω on the histograms are dependent on the the relative size ratiobetween ω and D, causing a slow convergence rate for small Ω.

3.1 Adding a temperature term β = 1

T

Sampling from the distribution p(I) using a Gibbs sampler does not easily enforcethe large scale geometric structure in the image. By using the Gibbs sampler onewill get a sample from the distribution, this includes both the stochastic and thegeometric structure of the image, however the stochastic structure will dominatethe result.

Adding an inverse temperature term β = 1T

to the distribution gives

p(I) =1

Z(Λ)exp−β

∑

α

∑

i

λαi Hα

i . (4)

In [6] we proposed a two step method to reconstruct both the geometric andstochastic part of the missing region Ω:

1. Cooling: By sampling from (4) using a fixed small temperature T value,structures with high probability will be reconstructed, while structures withlow probability will be suppressed. In this step large geometric structureswill be reconstructed based on the model p(I).

2. Heating: By sampling from (4) using a fixed temperature T ≈ 1, the texturecomponent of the image will be reconstructed based on the model p(I).

In the first step the geometric structure is reconstructed by finding a smoothMAP-like solution and in the second step the texture component is reconstructedby adding it to the large scale geometry.

In this paper we propose a novel variation of the above discussed method.We consider two cooling schemes and a gradual heating scheme which can beconsidered as the inverse of simulated annealing.

3.2 Cooling - the ICM solution

Finding the MAP solution by simulated annealing is very time consuming. Onealternative method is the Iterated Conditional Modes (ICM) algorithm. By let-ting T → 0 (or equivalently letting β → ∞) the conditional distribution (3) willbecome a point distribution. In each step of the Gibbs sampling one will set thenew intensity for a site (x, y)i to

Inewi = argmax

Ii

p(Ii | INi) . (5)

Fig. 1. From top left to bottom right: a) the image containing a damaged region b)the ICM solution c) the fast cooling solution d) adding texture on top of the fastcooling solution by heating the distribution e) total variation (TV) solution and f) thereconstructed region in context (can you find it?).

This is a site-wise MAP solution (i.e. in each site and in each step the mostlikely intensity will be selected). This site-wise greedy strategy is not guaran-teed to find the global MAP solution for the full image. The ICM solution issimilar but not identical to the high β sampling step described in [6]. The ICMsolution depends on initialisation of the unknown region Ω. Here we initialise bysampling pixel values identically and independent from a uniform distributionon the intensity range.

3.3 Cooling - Fast cooling solution

The MAP solution for the inpainting is the most likely reconstruction given theknown part of the image D\Ω,

IMAP = arg maxIi∀(x,y)i∈Ω

p(I | I(D\Ω), Λ) . (6)

Simulated annealing can be used for finding the MAP solution. Replacingβ in (4) with an increasing (decreasing) sequence βn called a cooling (heating)scheme. Using simulated annealing one starts to sample using a high temper-ature T and slowly cooling down the distribution (4) by letting T → 0. If βn

is increasing slowly enough and letting n → ∞ then simulated annealing will

find the MAP solution ( see e.g. [9, 10, 14]). Unfortunately simulated annealingis very time consuming.

To reconstruct Ω, the true MAP solution may not be needed, instead asolution which is close to the MAP solution may be enough. We therefore adopta fast cooling scheme, that does not guarantee the MAP solution. The goal isto reconstruct the geometric structure of the image and suppress the stochastictexture.

The fast cooling scheme used in this paper is defined as (in terms of β)

βn+1 = C+ · βn (7)

where C+ > 1.0 and β0 = 0.5.

3.4 Heating - Adding texture

The geometric structures of the image will be reconstructed by sampling usingthe cooling scheme. Unfortunately the visual appearance will be too smooth,and the stochastic part of the image needs to be added.

The stochastic part should be added in such a way that it does not destroythe large scale geometric part reconstructed in the previous step. This is done bysampling from the distribution (4) using a heating scheme similar to the coolingscheme presented in previous section and using the solution from the coolingscheme as initialisation.

The heating scheme in this paper is

βn+1 = C− · βn (8)

where C− < 1.0 and β0 = 25.By using a decreasing βn, value finer details in the texture will be reproduced,

while coarser details in the texture will be suppressed.

4 Results

Learning the FRAME model p(I) is computational expensive, therefore onlysmall image patches have been used. Even for small image patches the optimi-sation times are at least a few days. After the FRAME model has been learnt,inpainting can be done relatively fast if Ω is not to large.

The dynamic range of the images have been decreased to 11 intensity levelsfor computational reasons. The images that have been selected includes bothlarge scale geometric structures as well as texture.

The delta filter, 16 Gabor filters and 18 scale space derivative filters havebeen used in all experiments and 11 histogram bins have been used for all filters(see section 2.1 for a discussion).

In the cooling scheme (7), we use β0 = 0.5, C+ = 1.2 and the stoppingcriterion βn > 25 in all experiments. In the heating scheme (8), we use β0 =25,C− = 0.8 and the stopping criterion βn < 1.0.


Each figure contains an unknown region Ω of size 30 × 30 that should bereconstructed. Figure 1 contains corduroy images, figure 2 contains birch barkimages and figure 3 wood images. Each figure contains the original image withthe damaged region Ω with initial noise, the ICM and fast cooling solutions andthe solution of a total variation (TV) based approach [1] for comparison.

The ICM solution reconstruct the geometric structure in the corduroy, butfails to reconstruct the geometric structure in both the birch and the wood im-ages. This is due to the local update strategy of ICM, which makes it very sen-sitive to initial conditions. If ICM starts to produce wrong large scale geometricstructures it will never recover.

The fast cooling solution on the other hand seem to reconstruct the geometricstructure in all examples and does an even better job than the ICM solution forthe corduroy image. The fast cooling solutions are smooth and have suppressedthe stochastic textures. Because of the failure of ICM we only include results onheating based on the fast cooling solution.

The results - image d) - after the heating are less smooth Ω’s, but it is stillsmoother than I\Ω. The total variation (TV) approach produce a too smoothsolution even if strong geometric structures are present in all example.


5 Conclusion

Using FRAME to learn a probability distribution for a type of images givesa Gibbs distribution. The boundary condition makes it hard to use the learntGibbs distribution as it is for inpainting; it does not enforce large scale geometricstructures strongly enough. By using a fast cooling scheme a MAP-like solutionis found that reconstruct the geometric structure. Unfortunately this solution istoo smooth and does not contain the stochastic texture. The stochastic texturecomponent can be reproduced by sampling using a heating scheme. The heatingscheme adds the stochastic texture component to the reconstruction and decreasethe smoothness of the reconstruction based on the fast cooling solution.

A possible continuation of this approach is to replace the MAP-like step witha partial differential equation based method and a natural choice is the GibbsReaction And Diffusion Equations (GRADE) [15, 16], which are build on theFRAME model.

We decompose an image into a geometric component and a stochastic com-ponent and use the decomposition for inpainting. This is related to Meyer’s [17,18] image decomposition into a smooth component and a oscillating component(belonging to different function spaces). We find it interesting to explore thistheoretic connection with variational approaches.

Acknowledgements

This work was supported by the Marie Curie Research Training Network: Vi-siontrain (MRTN-CT-2004-005439).

References

1. Chan, T.F., Shen, J.: Variational image inpainting. Communications on Pure andApplied Mathematics 58 (2005)

2. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In:Proceedings of SIGGRAPH ’01, Los Angeles, California, USA (2001)

3. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: IEEEInternational Conference on Computer Vision, Corfu, Greece (1999) 1033–1038

4. Bonet, J.S.D.: Multiresolution sampling procedure for analysis and synthesis oftexture images. In: Computer Graphics, ACM SIGGRAPH (1997) 361–368

5. Bertalmio, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and textureimage inpainting. IEEE Transcations On Image Processing 12(8) (2003) 882–889

6. Gustavsson, D., Pedersen, K.S., Nielsen, M.: Geometric and texture inpainting bygibbs-sampling. In: SSBA07. (2007)

7. Zhu, S.C., Wu, Y.N., Mumford, D.: Filters, random fields and maximum entropy(frame): To a unified theory for texture modeling. International Journal of Com-puter Vision 27(2) (1998) 107–126

8. Zhu, S.C., Wu, Y.N., Mumford, D.: Minimax entropy principle and its applicationto texture modelling. Neural Computation 9(8) (1997) 1627–1660

9. Geman, S., Geman, D.: Stochastic relaxation, gibbs distribution, and the bayesianrestoration of images. IEEE Transaction PAMI (6) (1984) 721–741

10. Winkler, G.: Image Analysis, Random Fields, and Markov Chain Monte CarloMethods. Number 27 in Stochastic Modelling and Applied Probability. Springer-Verlag (2006)

11. Bigun, J.: Vision with Direction - A Systematic Introduction to Image Processingand Computer Vision. Springer-Verlag (2006)

12. Jain, A.K., Farrokhnia, F.: Unsupervised texture segmentation using gabor filters.Pattern Recogn. 24(12) (1991) 1167–1186

13. ter Haar Romeny, B.M.: Front-End Vision and Multi-Scale Image Analysis: Multi-Scale Computer Vision Theory and Applications, written in Mathematica. Vol-ume 27 of Computional Imaging and Vision. Kluwer Academic Publishers (2003)

14. Liu, J.S.: Monte Carlo Strategies in Scientific Computing. Springer Series inStatistics. Springer-Verlag (2004)

15. Zhu, S.C., Mumford, D.: Prior learning and gibbs reaction-diffusion. IEEE Trans-action on Pattern Analysis and Machine Intelligence 19(11) (1997) 1236–1250

16. Zhu, S.C., Mumford, D.: Grade: Gibbs reaction and diffusion equation - a frame-work for pattern synthesis, denoising, image enhancement, and clutter removal.IEEE Trans. PAMI 19(11) (1997) 1627–1660

17. Meyer, Y.: Oscillating Patterns in Image Processing and Nonlinear EvolutionEquations: The Fifteenth Dean Jacqueline B. Lewis Memorial Lectures. AmericanMathematical Society (AMS), Boston, MA, USA (2001)

18. Aujol, J.F., Gilboa, G., Chan, T., Osher, S.: Structure-texture image decompo-sition — modeling, algorithms, and parameter selection. International Journal ofComputer Vision 67(1) (2006) 111–136

GEOMETRIC AND TEXTURE INPAINTING BY GIBBS SAMPLING

David Gustavsson

IT-University of [email protected]

Kim S. Pedersen , Mads Nielsen

University of CopenhagenDIKU

kimstp,[email protected]

ABSTRACT

This paper discuss a method suitable for inpainting both largescale geometric structures and more stochastic texture com-ponents. Image inpainting concerns the problem of recon-structing the intensity contents inside regions of missingdata.Common techniques for solving this problem are methodsbased on variational calculus and based on statistical methods.Variational methods are good at reconstructing large scalege-ometric structures but have a tendency to smooth away tex-ture. On the contrary statistical methods can reproduce tex-ture faithfully but fails to reconstruct large scale structures.

In this paper we use the well-known FRAME (Filters,Random Fields and Maximum Entropy) for inpainting. Weintroduce a temperature term in the learned FRAME Gibbsdistribution. By sampling using different temperature in theFRAME Gibbs distribution, different contents of the imageare reconstructed.

We propose a two step method for inpainting using FRAME.First the geometric structure of the image is reconstructedbysampling from a cooled Gibbs distribution, then the stochas-tic component is reconstructed by sample from a heated Gibbsdistribution.

Both steps in the reconstruction process are necessary, andcontribute in two very different ways to the appearance of thereconstruction.

Index Terms— Inpainting, filling-in, image reconstruc-tion, texture, Gibbs distribution, FRAME, image decomposi-tion

1. INTRODUCTION

Image inpainting concerns the problem of reconstruction ofthe image contents inside a regionΩ. Based on the availablesurrounding image content one wants to reconstruct the re-gion. Some algorithms have reported excellent performancefor pure geometric structures (see [1] for a review), whileothers have reported excellent performance for pure textures([2, 3]).

The variational approach has shown to be very successfulfor geometric structures but have a tendency to produce a toosmooth solution.(See [1] for a review.)

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

18020 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

180

20 40 60 80 100 120 140 160 180

20

40

60

80

100

120

140

160

180

Fig. 1. To be able to reconstruct an image both the geometricand the stochastic (texture) part must be reconstructed. Thegeometric structures can be reconstructed by a MAP-like so-lution, while the stochastic component can be reconstructedafter the geometric structures have been added. Idea sketchfrom left to righta) the image to be reconstructed,b) a MAP-like solution containing the geometric structuresc) the recon-struction of the stochastic part after the geometric structureshave been added.

Bertalmio et. al. [4] propose a combined method in whichthe image is decomposed into a structure part and a texturepart, and different methods are used for filling the differentparts. The structure part is reconstructed using variationalmethod and the texture part is reconstructed by patch pasting.

Images contain information on different scales. Many im-ages contain a geometric structure part and a stochastic (tex-ture) part. The image is the composition of those two com-ponents. Ignoring the geometric structure of the image willresult in a noisy appearance of the image, while ignoring thestochastic part of the image will result in a too smooth image.This idea has previously been studied by Meyer [5] and byGuo, Zhu and Wu [6].

FRAME - Filters, Random Fields, and Maximum Entropyis a general method for finding a Gibbs distribution for a tex-ture that can be used for texture synthesis. Inpainting is dif-ferent from texture synthesis because of the extra constraintsthat the boundary conditions put on the reconstruction. Theboundary conditions put restrictions on howΩ can be recon-

structed. An inpainting contains both the reconstructed partI(Ω) and the original image partI(D\Ω); a small differencein the visual appearance ofI(Ω) is easier to discover viewedtogether withI(D\Ω).

In this paper we propose a two-step method for inpaintingusing the FRAME Gibbs distribution: First we cool down thedistribution to find a Maximum Posterior estimation (MAP)/ Iterated Conditional Mode (ICM)-like solution. Samplingfrom a Gibbs distribution using a low temperature will resultin a MAP / ICM-like solution. If the geometric structure iscontained in the boundary condition the region will containthe geometric structure. In the second step we heat up thedistribution to add the stochastic part of the texture. Samplingfrom a Gibbs distribution using a high temperature will resultin a less smooth and more stochastic appearance.

2. REVIEW OF FRAME

FRAME is a well known method for analyzing and reproduc-ing textures ([7, 8]). FRAME constructs a probability distri-bution -p(I) - for a texture from observed sample images.

Given a set of filtersFα(I) one computes the histogramof the filter responsesHα with respect to the filterα. Thehistograms are estimates of the marginal distribution. Giventhe marginal distributions for the sample images one wantsto find all distributions that have the same expected marginaldistributions, and among those find the distribution with themaximum entropy (the Maximum Entropy (ME) principle),that is the least committed distribution. This is a constrainedoptimization problem that can be solved using Lagrange mul-tipliers. The solution is

p(I) =1

Z(Λ)exp−

∑

i

∑

α

λαi Hα

i (1)

TheΛ = λαi - i is the number of bins for filterα - values

for a filterα gives information on how the different values forthat filter should be distributed. The relation betweenλα:sfor different filtersFα gives information on how the filtersare weighted.

An Algorithm for finding the distribution andΛ can befound in [8]. Once the distributionp(I) is known for a tex-ture it can be used for inference (analysis) and sampling (re-construction).

2.1. The Choice of Filter Bank

We have used two types of filter in our experiments: Gaborfilters and Scale Space derivative filters.

The Filters are:

• Delta (i.e the intensity)

• Gabor Filter - defined byGσ exp−iωx − iωy using8 orientations,ω = 0, π

4, π

2, 3π

4, π, 5π

4, 3π

2and 7π

4; 2

scalesσ = 1, 4, in total 16 Gabor filters has been used.

• Scale Space Derivative - using 3 scalesσ = 0.1, 1, 3

and 6 derivativesGσ, ∂Gσ

∂x, ∂Gσ

∂y, ∂2Gσ

∂x2 , ∂2Gσ

∂y2∂yand∂2Gσ

∂x∂y

HereGσ is the Gaussian function with standard deviationσ

Gσ(x, y) =1

2πσ2exp−

x2 + y2

2σ2 (2)

Which and how many filters should be used have a largeinfluence on the type of image that can be modelled. Thefilters must model the important visual appearance of the im-age at different scales. The filters must model both finer andcoarse scale structures.

Gabor filters are orientation sensitive and have been usedfor analyzing textures in a number of papers and are in generalsuitable for textures ([9, 10]).

The Scale Space Derivative is defined as

∂n

∂xn1∂yn2

(I ∗ Gσ) (3)

I is the image,Gσ is the Gaussian function (2) and∗ de-notes the convolution operator andn = n1 + n2.

It is well known from scale space theory that scale spacederivative filters capture structures at different scales.Byincreasingσ in the Gaussian function, finer details are sup-pressed, while coarse structures are enhanced. By using thefull scale-space both fine and coarse scale structures will becaptured.

2.2. Reconstruction

Once the distributionp(I) is learned, it is possible to use aGibbs sampler to sample from it.I is initialized randomly (orin some other way based on prior knowledge). We assumethat image domainD ⊂ R2 form a discrete lattice. A sitexi ∈ D is randomly picked and the intensity atxi is updatedaccording to the conditional distribution ([11] )

p(I(xi)|I(x−i)) (4)

x−i = D\xi. By sampling from the conditional distribu-

tion, I will be a sample from the distributionp(I).

3. USING FRAME FOR INPAINTING

When using FRAME for inpainting, only sites inside a dam-aged regionΩ ⊂ D are reproduced. A sitexi ∈ Ω is ran-domly picked and updated by sampling from the conditionaldistribution, given in 4. If the sitexi is close to the boundary∂Ω then the filters get support from sites insideΩ and outsideΩ (i.e. from sites inD\Ω). The sites outsideΩ are knownand fixed, and are boundary conditions for the inpainting.

We present a two-step method for inpainting using FRAME.In the first step the geometric structure will be constructedandin the second step the texture will be added. In both steps the

learned FRAME Gibbs distribution is used to reconstruct thedifferent parts.

3.1. Cooling - MAP/ICM solution

Sampling using a Gibbs sampler from the distributionp(I)does not enforce the geometric structure in the image. Byusing the Gibbs sampler one will get a sample from the dis-tribution, this includes both the stochastic and the geometricstructure of the image. Instead of adding both the stochasticand geometric part at the same time one could first add thegeometric part and then the stochastic part which may lead tofaster convergence. Adding an temperature1

Tto the distribu-

tion gives

p(I) =1

Z(Λ)exp−

1

T

∑

α

∑

i

λαi Hα

i (5)

here0 < T ≤ 1 andT = 1 corresponds to the leanedFRAME distribution (1). By sampling using different temper-aturesT , different image contents will either be suppressed orenhanced. In each iteration the Gibbs sampler set the intensityin a sitexi by draw an intensity from the conditional distribu-tion (4).

Sampling from the distribution (5) using a smallT value(using the conditional distribution (4)) will suppress intensi-ties with low (conditional) probabilities, and intensities withhigh (conditional) probabilities will have even higher (con-ditional) probabilities. T = 0 gives the so-called IteratedCondition Modes (ICM) solution ([11]). ICM is a site wisegreedy strategy and the intensity with the highest probabilityis selected in each iteration (or with probability1

kwherek is

the number of maximum in the conditional distribution).Sampling using a low temperature will generate a smoother

solution because of site wise greedier behavior. Values withlow probability will now have even lower probability, whilevalues with high probability will now have even higher prob-ability. The geometric structures of the image will be re-produced, and the stochastic variation (texture) will be sup-pressed.

3.2. Heating - Adding the texture

The geometric structures of the image will be found by sam-pling from the cooled distribution. Unfortunately is the visualappearance too smooth, and the stochastic part of the imageneeds to be added. This can be done by heating the distribu-tion - increaseT - and sample using a Gibbs sampler from theheated distribution.

When a largerT value is used, the probability mass forthe conditional distribution (4) will be spread more uniformlyover the different intensity levels. IfT → ∞ then the proba-bility mass for the conditional distribution will be uniformlydistributed. By increasingT the conditional distribution ismoving away from the site wise greedy strategy.

10 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

10010 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100

10 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

10010 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100

Fig. 2. Inpainting inΩ 30×30 using different temperaturesT .From top left to bottom right: a) Initialization using uniformlydistributed noise, b) Low temperature phase using1

T= 20, c)

High temperature phase using1T

= 2 and d) TV solution.

SettingT = 1 is equal to sample from the learned FRAMEGibbs distribution (i.e. the distribution 1 and 5 are equal).Hence, by usingT = 1 one will get a stochastic componenthaving the same statistics as the surrounding image.

By sampling from the Gibbs distribution using a largeT

value will add the texture. How largeT should be depends onthe scale of the texture. By using a largeT value fine detailsin the texture will be reproduced, while coarse details in thetexture will be suppressed.

4. RESULTS

The delta filter, 16 Gabor filters and 18 scale space derivativefilters have been used in all experiments.(See section 2.1 fora discussion.)

It is our experience that inpainting by sampling from theFRAME Gibbs distribution (1) is not possible. Inpainting bysampling from 1 does not reconstruct the geometric structureof the image, and the inpainting is poor.

In figure (2) a regionΩ−30×30− in an image containingcorduroy has been reconstructed using the two steps. Imageb

contains the inpainting after sampling from the Gibbs distri-bution with a low temperature -1

T= 20. The size ofΩ com-

pared with scale of the geometric structure and the texture ontop of the geometric structure makes the reconstruction diffi-cult. Imagec contains the inpainting after sampling from theGibbs distribution with a higher temperature -1

T= 2. The

smoothness found in imageb decreases, and the reconstructedregionΩ is visually similar toI(D\Ω).

In figure (3) a regionΩ − 25 × 25− in an image con-taining wood has been reconstructed using the two steps. Im-agec) contains the inpainting after sampling from the Gibbsdistribution with a low temperature -1

T= 4.The geometric

50 100 150 200 250 300

50

100

150

200

250

30010 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100

10 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

10010 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100

Fig. 3. Inpainting in Ω 25 × 25 using different tempera-turesT . From top left to bottom right: a) Undamaged im-age, b)Initialization using uniformly distributed noise,c) Lowtemperature phase using1

T= 4 and d) High temperature

phase using1T

= 1.5

structure is hard reproduced because of its’ irregularity;sizeand orientation is varying over the image.Imaged containsthe inpainting after sampling from the Gibbs distribution witha higher temperature -1

T= 1.5.

5. CONCLUSION

Using FRAME to learn a probability distribution for a type ofimages gives a Gibbs distribution. Because of the boundarycondition one can not use this distribution as it is for inpaint-ing, it does not enforce geometric structures strongly enough.A temperature term can be added to the Gibbs distribution tocontrol the stochastic behavior of the distribution.

By sampling using a fixed low temperature in the Gibbsdistribution geometric structures will be constructed, whiletextures will be suppressed. The result of inpainting usingalow temperature is as smooth reconstruction.

The stochastic (texture) part of the image is composedby sampling from the Gibbs distribution using a fixed hightemperature. By sampling using a fixed high temperature thestochastic part will be added.

Both steps in the reconstruction process are necessary, andcontribute in two very different ways to the appearance of thereconstruction.

6. ACKNOWLEDGEMENT

This work was supported by the Marie Curie Research Train-ing Network: Visiontrain (MRTN-CT-2004-005439).

7. REFERENCES

[1] Tony F. Chan and Jianhong Shen, “Variational image in-painting,” Communications on Pure and Applied Math-ematics, vol. 58, Februari 2005.

[2] A. A. Efros and W. T. Freeman, “Image quilting fortexture synthesis and transfer,” inProceedings of SIG-GRAPH ’01, Los Angeles, California, USA, August2001.

[3] J. S. De Bonet, “Multiresolution sampling procedure foranalysis and synthesis of texture images,” inComputerGraphics. ACM SIGGRAPH, 1997, pp. 361–368.

[4] Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro,and Stanley Osher, “Simultaneous strurcture and tex-ture image inpainting,” IEEE Transcations On ImageProcessing, vol. 12, no. 8, pp. 882–889, August 2003.

[5] Yves Meyer, Oscillating Patterns in Image Process-ing and Nonlinear Evolution Equations: The FifteenthDean Jacqueline B. Lewis Memorial Lectures, Amer-ican Mathematical Society (AMS), Boston, MA, USA,2001.

[6] Cheng en Guo, Song-Chun Zhu, and Ying Nian Wu,“Towards a mathematical theory of primal sketch andsketchability,” inProceedings of ICCV, 2003, vol. II,pp. 1228–1235.

[7] Song Chun Zhu, Ying Nian Wu, and David Mumford,“Minimax entropy principle and its application to tex-ture modelling,”Neural Computation, vol. 9, no. 8, pp.1627–1660, 1997.

[8] Song Chun Zhu, Ying Nian Wu, and David Mumford,“Filters, random fields and maximum entropy (frame):To a unified theory for texture modeling,”InternationalJournal of Computer Vision, vol. 27, no. 2, pp. 107–126,1998.

[9] Maria Petrou and Pedro Garcıa Sevilla,Dealing withTexture, Wiley, 2006.

[10] Anil K. Jain and Farshid Farrokhnia, “Unsupervised tex-ture segmentation using gabor filters,”Pattern Recogn.,vol. 24, no. 12, pp. 1167–1186, 1991.

[11] Gerhard Winkler,Image Analysis, Random Fields, andMarkov Chain Monte Carlo Methods, Number 27 inStochastic Modelling and Applied Probability. Springer-Verlag, 2006.