Alpha Matte Estimation of Natural Images UsingLocal and Global Template Correspondence

Muhammad Sarim, Adrian Hilton and Jean-Yves GuillemautCentre of Vision, Speech and Signal Processing

University of SurreyGuildford, GU2 7XH, United Kingdom.

Emails: m.farooqui@surrey.ac.uk, a.hilton@surrey.ac.uk and j.guillemaut@surrey.ac.uk

Abstract—Natural image matting is an interesting and difficultproblem of computer vision because of its under-constrainednature. It often requires a user interaction, a trimap, to aidthe algorithm in identifying the initial definite foreground andbackground regions. Current techniques use local or global imagestatistics of these definite regions to estimate the alpha matte forthe undefined region. In this paper we propose a novel non-parametric template correspondence approach to estimate thealpha matte. This technique alleviates the problem of previousparametric algorithms that rely solely on colour informationand hence are unable to exploit the image structure to theiradvantage. The proposed technique uses global and local templatecorrespondence, to the definite know regions, to construct thebackground and foreground layers. Once the foreground andbackground colours are estimated, the final alpha matte iscomputed. According to the quantitative analysis against theground truth, the proposed algorithm outperforms the currentstate-of-the-art parametric matting techniques.

I. INTRODUCTION

Digital image matting is a well studied problem of computervision. It is a process of extracting a foreground objectalong with its pixel-wise blending proportion from an image.Once the foreground colour and the blending proportion areestimated, the foreground object can seamlessly be compositedwith a desired background. Recent advancement in imagingtechnology and increasing demand for special effects in themedia industry fueled the research to develop complex androbust algorithms to estimate the alpha matte of naturalimages. An image can be thought as a composite of twolayers, the foreground and background. These two layers werefirst linearly formulated according to their blending proportion,alpha, by Porter and Duff [1] as

C = αF + (1− α)B. (1)

The equation is known as compositing equation, where, Cis the composite image while F and B are the foregroundand background layers and alpha (α) defines their blendingproportion. Alpha can take any value from 0 to 1, α = 0and α = 1 defines the pure background and foreground layerwhile any fractional alpha value gives the blending ratio of theforeground to background layer. The alpha values representedas an image lattice are called the alpha matte of an image. Tocompute the alpha matte from (1) we have to first estimatethe foreground and background layers, F and B respectively.The compositing equation is clearly under-constrained as the

only known variable is the composite image C. If we takeRGB image we have three equations, corresponding to RGBchannels, to solve for seven unknowns. To obtain a solutionof (1), constraints are required to define the foreground andbackground layers. Smith and Blinn in their seminal paper [2]explains how to get a trivial solution of (1) in studio envi-ronment by using a single or multiple homogeneous knownbackground colours. If a single uniform background colour,typically blue or green, is used we have to make an extraassumption that the foreground colour distribution is exclusiveto the background colour. In natural images the foreground andbackground colour is arbitrary and we can not impose suchconstraints. The algorithms that are used for natural imagematting, initially require a user interaction to identify the defi-nite foreground, background and unknown region in the image.This user input is called a trimap. The definite foreground,background and unknown regions are represented by white,black and gray colour respectively on the trimap lattice. Atypical trimap along with the original image, estimated alphamatte and the new composite image are shown in Fig 1. Oncethe trimap is provided techniques like [3], [4], [5], [6], [7], [8]fits a parametric model, normally a mixture of Gaussian, tothe local or global known foreground and background region.These models are then used to estimate the foreground andbackground colour for pixels in the unknown region and hencetheir alpha value from (1). Recently affinity based techniqueslike [9], [10], [11] gain popularity because they overcome themain limitation, of colour sample misclassification, in samplebased approaches. These techniques exploit local affinities toestimate the alpha matte in a propagation manner from theknown regions.

In this paper we present a novel non-parametric templatecorrespondence approach to natural image matting. The tech-nique not only exploits the spatially preserved colour informa-tion but also uses local image features and textures. Using thisadditional information available, our algorithm demonstratedbetter performance than other state-of-the-art parametric mat-ting techniques which only rely on colour information. Be-cause of the additional information used, our approach is ableto avoid the strong local smoothness assumptions made byprevious matting algorithms.

(a) Original image (b) trimap (c) estimated alpha matte (d) new composite

Fig. 1. Images are taken from the data-set provided by [10].

II. RELATED WORK

Since natural images have no constraints on the foregroundand background appearance and their relative colour distribu-tion, the matting process is initialised by a user constructedtrimap. Trimap is the aid provided to the algorithm to initiallyidentify the definite foreground and background regions in animage. The techniques like [4], [5], [8] fits the statistical modelof mixture of Gaussian to the local known regions. Thesemodels are later used to build the complete foreground andbackground layers and then estimate the blending proportionα using (1). Ruzon and Tomasi [8] modeled the local knownregion as mixture of isotropic Gaussians. The final alphamatte was estimated by using these isotropic clusters. Theintensity variations in natural images are considered in [5]and anisotropic Gaussians are used to model the local regionsmore precisely. Principal component analysis is then used toestimate the foreground and background colour for all thepixels in the unknown region and hence the alpha matte. In [4]these local anisotropic mixture of Gaussians are constructedby spanning a circular window through the unknown region inan onion skin manner. The already estimated foreground andbackground colour of the unknown pixels are also consideredin building the model. The matting problem is then formulatedin a Bayesian framework to estimate the final alpha matte.Since these approaches use local regions to sample colour forthe unknown region, to obtain a good alpha matte, a precisetrimap is necessary. To avoid this requirement [6], [7] usedglobal known regions to build the Gaussian mixture models.Alpha matte is then computed using these global models.Sample based algorithms have a fundamental limitation ofsample misclassification. Approaches like [9], [10], [11] usedlocal affinities to overcome this problem. Local smoothnessassumption is used in [9] to fit a linear model to the foregroundand background colours. This model provides a closed formsolution for blending proportion α. Poisson matting [10]assumed that the intensity variation in local foreground andbackground region is smooth. The algorithm solve the Poissonform of (1) using matte gradient filed to estimate the alphamatte. The technique suffers if the local intensity variationis not smooth. In [11] optimised colour samples and affinitysimilar to [9] are used to obtain a matting energy function interms of α. The global minimisation of this energy function

yields the final alpha matte. Affinity based approaches tend toaccumulate small errors because of their propagation behaviorin estimating the alpha values.

The template correspondence approach presented in thispaper uses nonparametric statistics, which have previouslybeen used successfully to represent spatially preserved localimage colours, features and textures for inpainting [12], [13]and view interpolation [14].

III. ALPHA MATTE ESTIMATION USING TEMPLATECORRESPONDENCE

Our technique can be divided into three parts: (1) back-ground layer construction using global inpainting, (2) fore-ground layer estimation using local foreground template cor-respondence and (3) the final estimation of alpha matte.

A. Background layer construction

We used global inpainting technique similar to [13] to con-struct the background layer. Given a trimap, the backgroundlayer is grown inwards from the definite background boundaryto completely cover the unknown region. The algorithm startsby building a global background template space, Bg , bylocalising a square patch of size m at every pixel in thedefinite background region identified in the trimap as black.The pixels near the background boundary are excluded fromtemplate space construction because the patch localised atthese pixels also contain the unknown pixels. All the templatespresent in the template space Bg only contain the definitebackground pixels. To construct the background inwards, letus localise a patch up at an unknown contour pixel p as shownin Fig 2(a). The patch up should be dimensionally consistentwith the templates in Bg for comparison. Since the patch up

consists of both the background and the unknown pixels, apartial comparison between up and the templates in Bg isrequired. To ensure such comparison, a binary mask BMb

is constructed to mask-out the unknown pixels in the templateup. The logical true, 1, in the binary mask BMb correspondsto the background pixels in the template up while the logicalfalse, 0, points to the unknown pixels in up. The binary maskBMb can be visualise as a template, consistent with up, asshown in the Fig 2(a) having 1′s for the blue pixels and 0′sfor the gray pixels. Let us define a function ∆ (M,N,BM)as a sum of absolute difference between two, dimensionally

(a) (b) (c) (d)

Fig. 2. Background layer construction scheme: (a) template up, (b) most similar template bq , (c) pixel transferred from bq to up and (d)filled in background region.

consistent, square templates M and N . The logical true inthe binary mask BM associated with the template M identifythe pixels to be compared to compute the sum of absolutedifference. If the two patches have a dimension of m andlocalised at the Euclidean coordinates (xi, yi), and (xj , yj)respectively, the function can be written as

∆ (M,N,BM) =g∑

s=−g

h∑t=−h

{|M(xi + s, yi + t)−

−N(xj + s, yj + t)| . BM (xi + s, yi + t)}

(2)

Where, g = h = ± (m− 1) /2. The Unknown region is filledin by the background layer by finding the most similar patchin the global background template space Bg to the templateup. Let us denote the Euclidean coordinated of the unknownpixel p by (xp, yp). Now the most similar patch bq can befound by

bq = argminbi∈Bg

1np

∆(up, bi,BMb

). (3)

Equation (3) gives the template bq that has the minimumnormalised sum of absolute difference from the template up

in colour space as shown in Fig 2(b). The binary mask BMb

ensures the comparison of only the background pixels in thepatch up to the corresponding pixels in the patch bi ∈ Bg . Toensure the cost is comparable the sum of absolute differenceis normalised by np, the total number of background pixelsin the template up. The unknown region is filled in by thebackground layer by transferring all the pixels in the templatebq that corresponds to the unknown pixels in the patch up asshown in Fig 2(c,d). The process is iterated until the wholeunknown region is filled in to form the estimated backgroundlayer B̃. There is no need to estimate the background colourfor the definite foreground region of the trimap (white) becausethe background layer is completely occluded by the foregroundlayer in this region.

B. Foreground layer construction

The foreground layer for the unknown region of the trimapis constructed in a similar fashion as in the section III-A. Since

there is always a strong correlation exist between the localpixels, to estimate the foreground layer unlike global approachin section III-A, we utilised local template correspondences.This local approach also reduces the computational burden.Let us take a pixel p in the unknown region, the algorithmstarts by constructing the local foreground template space F l

for this pixel. The local circular foreground region, of radiusRp, is identified by the two Euclidean distance vectors rpand ri as Rp = rp + ri. The minimum spatial distance

Fig. 3. The local circular foreground region for pixel p, the innerarc has a radius of ri and the outer arc has a radius of Rp.

of pixel p from the foreground boundary is denoted by rpwhile ri is the predefined constant distance vector. The localforeground region for pixel p is shown in Fig 3, the innerarc has a radius of ri while the outer arc has a radius ofRp. The Euclidean distance vector ri ensures to define asufficient search region for the pixels lying near the foregroundboundary having very small distance from the foregroundedge. The distance rp is introduce in the final search dimensionRp to define a large search region for the pixels lying nearthe background boundary having weak correlation with theforeground boundary pixels. The algorithm constructs thelocal foreground template space F l, similar to the global

Fig. 4. The pixels identified as background and undefined arerepresented by red and blue respectively.

background template space Bg , by localising a square templateof size m at every pixel in the local foreground region. Thenext step is to identify the blended and the pure foregroundpixels in the unknown region to avoid the comparison of thepure background pixels with the definite foreground pixelsin the template space F l. The identification can be doneby subtracting the constructed background layer from thecomposite image C. The subtraction is performed patch-wiserather than pixel-wise to avoid the misclassification of weaklyblended foreground pixels like fine hairs and strands. Thesubtraction is only done for the pixels corresponding to theunknown region of the trimap. The subtracted layer S is givenby

S(x, y) =g∑

s=−g

h∑t=−h

∣∣∣(C(x+ s, y + t)− B̃(x+ s, y + t)∣∣∣ .(4)

Where, S(x, y) defines the sum of absolute difference, incolour space, between the square patches of size m localisedat the Euclidean coordinates (x, y) in the composite imageC and the background layer B̃ while g = h = (m− 1) /2.To identify the blended and the pure foreground pixels in theunknown region, a predefined distance threshold ε, in colourspace, is applied to the subtracted layer S as

pixel(x, y) =

background if, S(x, y) ≤ ε

undefined otherwise(5)

Where, pixel(x, y) represents a pixel at Euclidean coordinates(x, y) and undefined means a blended or a pure foregroundpixel in the unknown region of the trimap. The backgroundand undefined pixels are represented as red and blue in Fig4 respectively. Now the foreground layer is constructed forall the undefined pixels by using local foreground templatecorrespondence. Let us consider an undefined pixel p. Thealgorithm first localises a square template, up of size m atpixel p. A binary mask BMf associated with the templateup, similar to section III-A, is constructed. To avoid the falsecomparison, the background(red and black) pixels are replacedby 0′s and undefined(blue) pixels by 1′s in the template up to

form the binary mask BMf . The foreground colour for pixelp is estimated by finding the most similar template, fq , in thelocal template space F l as

fq = argminfi∈Fl

1n′

∆(up, fi,BMf

). (6)

Where, g = h = (m− 1) /2, the function ∆ has the samedefinition as in section III-A and n′ is number of undefinedpixels in the template up, used for normalising the sum ofabsolute difference. The foreground colour F (p) for pixel pis estimated by the colour of pixel q in the foreground region.The noise present in the foreground region could result in thefalse estimation of the foreground colour, F (p), of pixel ptherefore an additional robust estimation criteria is required.

1) Robust estimation criteria: Let us define a normalisedsum of absolute difference vector D, in colour space, for thetemplate up from the template space F l as

D(i) =1n′

∆(up, fi,BMf

). (7)

The distance vector D and the local foreground template spaceF l are sorted such that D(j) < D(j + 1). Now let us denotethe centre pixel colour of the n most similar patches from thesorted template space F l as {f c

1 , fc2 , ..., f

cn}. The foreground

colour F (p) for pixel p is estimated as the median colour ofthe aforementioned centre pixel colour vector i.e.

F (p) = µ1/2 {f c1 , f

c2 , ..., f

cn} . (8)

Where, µ1/2 defines the median. For the current paper we usedthree most similar patches in the template space i.e. n = 3. Theprocess is iterated for all the undefined pixels in the unknownregion of the trimap to form the estimated foreground layerF̃ .

C. Alpha matte estimation

Once the background and foreground layers are estimated,compositing equation can be reformatted to estimate the alphamatte as

α =C − B̃F̃ − B̃

. (9)

Where, C is the composite image while F̃ and B̃ are theestimated foreground and background layers. Once the alphamatte α is estimated, the foreground layer can seamlessly becomposited into a new background.

IV. RESULTS AND EVALUATION

We present a detailed qualitative and quantitative analysis ofour technique. For qualitative comparison we used two naturalimages as shown in Fig 5 and for quantitative analysis weused three composite images as shown in Fig 6. The imagesare taken from the data set provided by [11]. The compositeimages are formed by first capturing the foreground objectagainst multiple homogeneous backgrounds then a triangulartechnique proposed in [2] is used to extract the ground truthand the foreground layer. New background layer along withthe estimated foreground layer and the ground truth alpha

(a) Original (b) trimap (c) Hillman (d) Poisson (e) Closed form (F) Robust (g) Nonpara

Fig. 5. Natural images along with their trimaps and alpha matte estimated by different techniques. Images are taken from the data set of[11].

(a) Original (b) trimap (c) Hillman (d) Poisson (e) Closed form (F) Robust (g) Nonpara (h) ground truths

Fig. 6. Composite images along with their trimaps, alpha matte estimated by different techniques and ground truths. Images are taken fromthe data set of [11].

matte is used to form the final composite images of Fig 6. Forthis paper the aforementioned parameters of our algorithm arechosen as m = 5, the template size, and ri = 50 pixels,predefined local foreground radius, while the value of thebackground difference threshold ε is a variable that dependson the individual image. We compared our technique withfour other natural image matting algorithms: (1) Hillman [5](2) Global Poisson matting [10], (3) Closed form [9] and (4)Robust matting [11].

A. Qualitative evaluation

The foreground object in the natural images of Fig 5 arevery complex as they contain large hairs which are difficult tomatte. The first image in particular has a very large unknownregion because of the hairs. Hillman approach produced ac-ceptable result for a precise trimap but failed for the largeunknown regions as it is a sample based technique. GlobalPoisson matting also produced reasonable result but sufferedfrom segmentation inaccuracies where the local intensity vari-ations are not smooth and produced a fuzzy matte for largeunknown region. Closed form technique failed to extract agood matte for the fine hairs having the similar backgroundcolour distribution. Robust matting and our algorithm bothproduced good alpha mattes. The results of Robust matting is

slightly over smooth in the hairs where they have similar back-ground colour. Our algorithm demonstrated better performancein handling the fine hairs of similar background colour andthe estimated alpha mattes are free from visible segmentationinaccuracies. If the background colour distribution is distinctfrom the foreground as in the first two composite imagesof Fig 6. Closed form, Robust matting and Our algorithmproduced alpha mattes which are similar to the ground truth.Hillman and Poisson matting suffered the same limitations asin the natural images. All the previous matting algorithms usedin this paper failed to produce acceptable result for the lastcomposite image shown in Fig 6. The image is very complexboth in the foreground and background appearance becauseof fine hairs in the foreground object and presence of hightexture and complex features in the background. Our mattingalgorithm produced good alpha matte for this image proving itsability to handle highly complex images by exploiting the localimage features and textures beside colour information. Theestimated alpha matte does not have visible artifacts comparedto the ground truth.

B. Quantitative evaluation

For the quantitative analysis of the alpha mattes generatedby different algorithms for the composite images in Fig 6, we

Fig. 7. Mean absolute error present in the alpha mattes produced by differenttechniques for the three composite images. The initials of the technique is usedas the legend.

used mean absolute error MAE as the error measurement.Although the mean absolute error is not always correlated tothe visual quality of the alpha matte, it provides a sensiblequantitative comparison. The mean absolute error plot ofdifferent techniques is shown in Fig 7. It is clear from thegraph that the Poisson matting is severely affected by thelack of smoothness in the local foreground and backgroundregion while Hillman’s approach produced large error due tothe large unknown region and overlap of local foregroundand background colour distribution. The Closed form andRobust matting produced less error for the lion and tail imagebecause of their relatively simple background but sufferedwith large errors for the more complex bird image. Ourmatting technique produced mattes that have consistently smallerror and significantly outperforms the other state-of-the-artalgorithms for the highly complex bird image.

V. CONCLUSION

In this paper we have presented a novel natural image mat-ting technique based on template correspondence. The mainadvantage of using nonparametric approach is its ability toexploit not only the colour information but also the local imagetextures and features to better estimate the foreground andbackground layers. The qualitative and quantitative evaluationshows that our algorithm produced similar results to the otherstate-of-the-art techniques for relatively simple images. Forhighly textured and complex images, where state-of-the-artapproaches fail to produced acceptable results, our algorithmgenerated alpha mattes having very small error and no visibleartifacts. Future work will focus on optimizing the algorithmby developing and evaluating different robust matching criteriaand utilizing more local constraints.

REFERENCES

[1] T. Porter and T. Duff, “Compositing digital images,” in ACM SIGGRAPH’84: Proceedings of the 11th annual conference on Computer graphicsand interactive techniques, 1984, pp. 253–259.

[2] A. R. Smith and J. F. Blinn, “Blue screen matting,” in ACM SIGGRAPH’96: Proceedings of the 23rd annual conference on Computer graphicsand interactive techniques, 1996, pp. 259–268.

[3] A. Berman, A. Dadourian, and P. Vlahos, “Method of removing froman image the background surrounding a selected object.” U.S. Patent6,134,346, 2000.

[4] Y. Y. Chuang, B. Curless, D. H. Salesin, and R. Szeliski, “A bayesianapproach to digital matting,” in Proceedings of IEEE CVPR ’01, vol. 2,December 2001, pp. 264–271.

[5] P. Hillman, J. Hannah, and D. Renshaw, “Alpha channel estimation inhigh resolution images and image sequences,” in IEEE CVPR, 2001, pp.1063–1068.

[6] X. Bai and G. Sapiro, “Geodesic matting: A framework for fastinteractive image and video segmentation and matting,” Int. J. Comput.Vision, vol. 82, no. 2, pp. 113–132, 2009.

[7] J. Wang and M. F. Cohen, “An iterative optimization approach for unifiedimage segmentation and matting,” in ICCV ’05: Proceedings of the TenthIEEE International Conference on Computer Vision. Washington, DC,USA: IEEE Computer Society, 2005, pp. 936–943.

[8] M. A. Ruzon and C. Tomasi, “Alpha estimation in natural images,” inCVPR, June 2000, pp. 18–25.

[9] A. Levin, D. Lischinski, and Y. Weiss, “A closed form solution tonatural image matting,” Computer Vision and Pattern Recognition, IEEEComputer Society Conference on, vol. 1, pp. 61–68, 2006.

[10] J. Sun, J. Jia, C. Tang, and H. Y. Shum, “Poisson matting,” ACMTransactions on Graphics, vol. 23, no. 3, pp. 315–321, 2004.

[11] J. Wang and M. F. Cohen, “Optimized color sampling for robustmatting,” Computer Vision and Pattern Recognition, IEEE ComputerSociety Conference on, vol. 0, pp. 1–8, 2007.

[12] A. Efros and T. Leung., “Texture synthesis by non-parametric sampling.”in IEEE International conference on computer vision, 1999, pp. 1033–1038.

[13] A. Criminisi, P. Prez, and K. Toyama, “Object removal by exemplar-based inpainting,” Computer Vision and Pattern Recognition, IEEEComputer Society Conference on, vol. 2, pp. 721–728, 2003.

[14] A. Fitzgibbon, Y. Wexler, and A. Zisserman, “Image based rederingusing image based priors,” in International conference on computervision ICCV, 2003, pp. 1176–1184.