10
Image stitching with dynamic elements Alec Mills * , Gregory Dudek McGill University, Center for Intelligent Machines, 3480 University Street, Montreal, Que., Canada H3A 2A7 article info Article history: Received 19 October 2007 Received in revised form 6 January 2009 Accepted 2 March 2009 Keywords: Image stitching Mosaic Blending Computer vision SIFT Optimal seam selection Panorama Parallax abstract This paper presents a new combination of techniques to create pleasing and physically consistent image mosaics despite the presence of moving objects in the scene. The technique uses heuristic seam selection in the intensity and gradient-domains to choose which pixels to use from each image and then blends them smoothly to create the final mosaic. We demonstrate illustrative results obtained by comparing and contrasting our output with that obtained from four representative existing image mosaic systems. One of these is documented in the aca- demic research literature and three are commercially available products. The present algorithm gives comparable or superior results in all examples. Ó 2009 Elsevier B.V. All rights reserved. 1. Introduction Image mosaic synthesis has recently received substantial atten- tion in both the research literature as well as in the form of com- mercial applications [25]. Image mosaic or panorama synthesis refers to the combination of two or more images into a single com- posite image. Many algorithms today are capable of taking overlap- ping images of the same scene and quickly stitching them together to create a panorama in which the transitions between compo- nents from different images are not noticeable. These algorithms, however, usually have a number of requirements to produce satis- factory results: limited camera translation, limited lighting varia- tion, similar exposure settings between images and limited motion of objects in the scene. Unfortunately most algorithms fail to some degree when the scene exhibits substantial motion and the result often contains cut objects or heavy blurring. Many envi- ronments encompass moving objects and to produce a high quality result, moving objects must be correctly handled. In this work, we are concerned with computing a mosaic con- taining dynamic subjects and large-scale lighting variation. We do not concentrate on the rich problem of robust homography computation but rather employ a pre-existing method, SIFT [17]. A comparison of many modern feature-based techniques [19] found that SIFT outperformed the other available techniques in most tests. Conventional mosaic synthesis operates by selecting content from one image or another as a function of which side of a dividing line or seam a given pixel lies upon. This paper introduces a new method for selecting the seam by defining a heuristic error metric over pixels with respect to several image properties that define a good mosaic. We assume that the scene is composed of a largely static background with a limited number of moving objects that occupy a minority of the total set of pixels. Ideally, one might wish to explicitly identify and separate these objects and then assure that the image seam is selected to either fully include or exclude an object from the final mosaic, but never select only a fraction of an object for inclusion. Since object-based segmentation is diffi- cult to define unambiguously, we instead seek to avoid it and sug- gest heuristic functions that strongly penalize seam pixels that are likely to lie in the interior of moving objects. Although user speci- fied compositing hints might simplify the problem, a fully auto- mated solution is much simpler for the user than a solution including manual hints, thus that is the focus of this paper. A difficulty in creating the seam is that the images being mosa- icked may have been taken under different lighting conditions or even with different cameras, creating intensity differences be- tween pixels that contain projections of the same scene points. To cope with this, the images are corrected to have similar expo- sure parameters using a linear model of bias and gain. Then the seam is calculated based on a combination of differences in the intensity domain and in the gradient domain. If the exposure cor- rection is small, more weight is given to the intensity differences. If the correction is large, the linear approximation is likely to be a 0262-8856/$ - see front matter Ó 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2009.03.004 * Corresponding author. Tel.: +1 514 398 6319; fax: +1 514 398 7348. E-mail address: [email protected] (A. Mills). Image and Vision Computing 27 (2009) 1593–1602 Contents lists available at ScienceDirect Image and Vision Computing journal homepage: www.elsevier.com/locate/imavis

Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

Embed Size (px)

Citation preview

Page 1: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

Image and Vision Computing 27 (2009) 1593–1602

Contents lists available at ScienceDirect

Image and Vision Computing

journal homepage: www.elsevier .com/ locate / imavis

Image stitching with dynamic elements

Alec Mills *, Gregory DudekMcGill University, Center for Intelligent Machines, 3480 University Street, Montreal, Que., Canada H3A 2A7

a r t i c l e i n f o

Article history:Received 19 October 2007Received in revised form 6 January 2009Accepted 2 March 2009

Keywords:Image stitchingMosaicBlendingComputer visionSIFTOptimal seam selectionPanoramaParallax

0262-8856/$ - see front matter � 2009 Elsevier B.V. Adoi:10.1016/j.imavis.2009.03.004

* Corresponding author. Tel.: +1 514 398 6319; faxE-mail address: [email protected] (A. Mills).

a b s t r a c t

This paper presents a new combination of techniques to create pleasing and physically consistent imagemosaics despite the presence of moving objects in the scene. The technique uses heuristic seam selectionin the intensity and gradient-domains to choose which pixels to use from each image and then blendsthem smoothly to create the final mosaic.

We demonstrate illustrative results obtained by comparing and contrasting our output with thatobtained from four representative existing image mosaic systems. One of these is documented in the aca-demic research literature and three are commercially available products. The present algorithm givescomparable or superior results in all examples.

� 2009 Elsevier B.V. All rights reserved.

1. Introduction

Image mosaic synthesis has recently received substantial atten-tion in both the research literature as well as in the form of com-mercial applications [25]. Image mosaic or panorama synthesisrefers to the combination of two or more images into a single com-posite image. Many algorithms today are capable of taking overlap-ping images of the same scene and quickly stitching them togetherto create a panorama in which the transitions between compo-nents from different images are not noticeable. These algorithms,however, usually have a number of requirements to produce satis-factory results: limited camera translation, limited lighting varia-tion, similar exposure settings between images and limitedmotion of objects in the scene. Unfortunately most algorithms failto some degree when the scene exhibits substantial motion andthe result often contains cut objects or heavy blurring. Many envi-ronments encompass moving objects and to produce a high qualityresult, moving objects must be correctly handled.

In this work, we are concerned with computing a mosaic con-taining dynamic subjects and large-scale lighting variation. Wedo not concentrate on the rich problem of robust homographycomputation but rather employ a pre-existing method, SIFT [17].A comparison of many modern feature-based techniques [19]found that SIFT outperformed the other available techniques inmost tests.

ll rights reserved.

: +1 514 398 7348.

Conventional mosaic synthesis operates by selecting contentfrom one image or another as a function of which side of a dividingline or seam a given pixel lies upon. This paper introduces a newmethod for selecting the seam by defining a heuristic error metricover pixels with respect to several image properties that define agood mosaic. We assume that the scene is composed of a largelystatic background with a limited number of moving objects thatoccupy a minority of the total set of pixels. Ideally, one might wishto explicitly identify and separate these objects and then assurethat the image seam is selected to either fully include or excludean object from the final mosaic, but never select only a fractionof an object for inclusion. Since object-based segmentation is diffi-cult to define unambiguously, we instead seek to avoid it and sug-gest heuristic functions that strongly penalize seam pixels that arelikely to lie in the interior of moving objects. Although user speci-fied compositing hints might simplify the problem, a fully auto-mated solution is much simpler for the user than a solutionincluding manual hints, thus that is the focus of this paper.

A difficulty in creating the seam is that the images being mosa-icked may have been taken under different lighting conditions oreven with different cameras, creating intensity differences be-tween pixels that contain projections of the same scene points.To cope with this, the images are corrected to have similar expo-sure parameters using a linear model of bias and gain. Then theseam is calculated based on a combination of differences in theintensity domain and in the gradient domain. If the exposure cor-rection is small, more weight is given to the intensity differences. Ifthe correction is large, the linear approximation is likely to be a

Page 2: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

1594 A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602

poor approximation and gradient differences are given a higherweight.

Our algorithm performs the mosaicking on successive imagepairs in three steps, registration, seam selection and blending.

Fig. 1. Outline of the algorithm.

2. Related work

Image mosaic synthesis can be based on either global imagemeasurements or local features. In either case, these measure-ments are brought into correspondence with one another in a pro-cess called registration. A critical step that distinguishes mosaicsynthesis from stereo is the need to merge the images into a singlefinal result. This key process depends on defining a seam alongwhich the merge is defined. The seam can be made nearly invisibleby first correcting for exposure differences and then by using agood blending scheme.

Some of the earliest published work on image registration isthat of Barnea and Silverman in 1972 [5] and Lucas and Kanade[18].

In the field of exposure correction, the authors of [4] iterate be-tween direct measurement and a linear exposure correction to at-tempt to register the image. In [26], the image is divided intoblocks and each one is individually corrected then smoothed. In[2,15] stitching is done entirely in the gradient-domain to avoidexposure differences. In [7], gain compensation is performed bymultiplying the mean value of pixels in each image by a constantto minimize an error function.

After a set images have been registered, a method of blending orselecting pixels in the overlapping regions must be used. The sim-plest way is to average the pixels, but this leaves visible seamswith changes in lighting. Ghosting or parallax can also occur ifthe camera motion was not purely rotational or if subjects moveslightly between each exposure. Some methods designed to im-prove pixel selection are the median filter [14], center weighting[25], minimum likelihood, optical flow [23] and the over operation[22]. A possible solution to ghosting is to chose a seam in a moreintelligent manner and pick pixels from one photo or anotherbased on which side of the seam they fall. One possible seam is aVornoi diagram [27,21]. This can be combined with blending fora smoother transition [28]. All these methods fail when presentedwith moving objects that pass through the seam.

The solution to the dynamic object problem is to select a seambased on the contents of the image rather than simply splitting thestitch down the middle [20]. One algorithm for this is to place aseam along the edges of objects in the picture [24]. This fails if adynamic object is passing over one of the chosen edges at the timethe image was taken. A more modern approach is using regions ofdifference [26]. This technique identifies dynamic objects bychecking the source images to see where pixels differ by more thana threshold. Then, to create the mosaic, the object is chosen fromone of the images in a pseudo-random way. This algorithm workswell, but is easily thwarted by exposure differences. Another tech-nique with excellent results is [3]. This technique requires that theuser selects the relative importance of seam and difference andtherefore is not suitable for our goal of a fully automated solution.

The closest work to ours in this field is [9]. This paper uses Dijk-stra’s algorithm [10] to chose an optimal seam when the sourcepictures have moving objects. The seam is placed by first creatinga difference image of the two registered images, with pixels givenhigher values if they have larger intensity differences. The seam isthen the minimum cost path from one edge of the overlapping areato the other. The formula used by Davis to determine the differenceof pixels is also used here. Unfortunately, the algorithm cannothandle illumination changes in the images and therefore is onlyuseful for carefully controlled source images. The reason for this

is that the algorithm neglects to compensate for exposure differ-ences and does not use any blending techniques. The work in thispaper improves and corrects those omissions and increases thequality of the stitch by considering gradient information. Similarwork has been done in [1] and in the field of image quilting [11,16].

3. Algorithm overview

An overview of our algorithm is shown in Fig. 1. First the regis-tration transformation between images (i.e., a homography) iscomputed, followed by a seam computation and finally a blendingof the two images to produce a final result.

A good seam is a seam which divides the final mosaic into re-gions taken from the source images such that few discontinuitiesoccur along the boundary. An optimal seam falls along the pathof lowest intensity in the difference image, producing the mini-mum discontinuity in the final mosaic. This path will usually avoidobjects in motion and regions where lens distortion or parallax ef-fects are noticeable.

An example of input images are shown in Fig. 2.

3.1. Registration

The registration step uses feature-based correspondence to esti-mate a transformation.

1. Each image has SIFT features identified. These features include ascale, rotation and location on each image, as well as adescriptor.

2. The features are matched by descriptor between images usingan approximate nearest neighbor algorithm.

3. The best consistent set of matches under a projective transfor-mation is found using RANSAC [12].

4. The consistent features are used to find the projective transform[13] which will map one image onto the other.

Page 3: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

Fig. 2. Example pair of input images.

A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602 1595

3.2. Optimal seam selection

In the overlapping region, we now decide which image to selecteach pixel from. A seam is found cutting the overlapping regioninto two uneven areas, with each region drawing pixels form a dif-ferent image.

3.2.1. Exposure correctionTruly constant scene illumination is rarely achieved in practice

and automatic gain control in the imaging system can lead toapparent changes in lighting. It is therefore necessary to correctfor exposure differences. To do this, we use the assumption thatthe reflectance properties of the scene remain constant in the re-gion of overlap. This allows us to compute the average luminanceincident on the photodetector over the overlapping region, and tonormalize the input images to keep this constant.

To make the adjustment, a linear approximation is used.

pixel1 ¼ a � pixel2 þ b

The gain, a and bias, b, between the images are found using a mod-ified version of RANSAC. The algorithm chooses only the static pix-els to calculate exposure correction, as the dynamic regions wouldskew the information. RANSAC inliers for the exposure correctionare shown in Fig. 3.

3.2.2. IntensityTo select the seam, some measure of similarity must be calcu-

lated for the regions of the image. This is done on a pixel-by-pixelbasis using both intensity differences and gradient differences. Theintensity differences ðdIÞ between the pixels of the two images arecalculated, as done by Davis [9] by taking the absolute value of thedifference of each pixel divided by their maximum.

Fig. 3. Pixels chosen to calculate gain. Non-overlapping pixels are in red, RANSACoutliers are in blue and RANSAC inliers are in green. Dynamic objects are mostlyexcluded.

dIij ¼

absðIm1ij � Im2ijÞmaxðIm1ij; Im2ijÞ

After calculating the difference, a small value is subtracted fromevery pixel (to a minimum of 0) and then normalized. Doing this re-moves noise from the pixels and encourages the algorithm to chosea longer path to avoid an object, rather than simply the shortestnoisy path.

The intensity differences of the sample images are shown inFig. 4.

3.2.3. GradientIf the exposure differences between the two images are signifi-

cantly non-linear, the exposure correction will not work. In thiscase, the intensity differences will not be a good measure of simi-larity and a different metric is needed. To obtain this metric, theimages are converted to the gradient-domain. The differences ofeach pixel ðdrÞ are found and noise is removed using the sametechnique as before. The gradient differences of the sample imagesare shown in Fig. 5.

3.2.4. Synthesis of the techniquesThe soft constraints resulting from each of these factors is com-

bined using a weighted sum

Dij ¼ w1dI þw2d

r

Fig. 4. Intensity differences between the two images. Black indicates no difference,white is a large difference.

Page 4: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

Fig. 5. Gradient differences between the two images. Black indicates no difference,white is a large difference.

Fig. 7. Combined difference values for the sample images. Black pixels have lowdifference.

Fig. 8. Section of image 1 to the right of the seam and section of image 2 to the leftof the seam. The images have been aligned with a slight separation for illustrativepurposes.

1596 A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602

As is illustrated in Fig. 6, weight selection is an important problem.In this example, a greater weight on the gradient ðw2Þ gives the de-sired result, whereas in other examples the opposite may be true.The image in Fig. 11, for example, fails to find a correct seam ifthe value of w2 is increased (not shown).

The optimal weights for combining these functions are depen-dent on numerous factors that are difficult to evaluate. However,as the illumination differences between the source images in-crease, more weight is needed on the gradient image to obtainthe desired result. The increased weight is likely due to the linearapproximation used in exposure correction, which becomes lessvalid as the differences become more pronounced. Fig. 10, showsan example of a poor linear approximation. Therefore weightsare calculated such that w2 becomes larger as the gain deviatesfrom unity and the bias deviates from null. These weights will bedifferent for every image pair and are calculated automaticallybased on the exposure correction.

� w2 ¼ ðabsðlnðaÞÞ þ absðbÞÞ2 is the gradient difference weight. Inthe presence of a large exposure correction, the linear approxi-mation becomes less accurate and the intensity image is lessreliable. Therefore, more weight is given to the gradient image.

� w1 ¼ 1�w2 is the weight of the intensity difference. If the cal-culated value is negative, a weight of 0 is assigned to w1.

In order to assure the seam does not cut through an accidentallow intensity path, it is prudent to blur the weights of the pixels.An accidental path occurs when the colors in a dynamic object hap-pen to fall in such a way that there exists a path of nearly identical

Fig. 6. Two stitched photos with the weights given to t

pixels. Usually these paths are surrounded by pixels with highintensity differences and applying a blur will eliminate them. Inaddition the pixels near the exterior of the image are given higherintensities (inflated) to discourage the optimal seam to trace theedge of the difference image. The amount of inflation can signifi-cantly change the location of the seam and it is unclear how todetermine the best value.

The difference values for the sample images are shown in Fig. 7.

3.3. Optimal seam selection

As we have noted, a good seam is one that is not visible. Thissuggests that a seam should either span regions where it willeither not introduce a brightness change, or where an induced

he gradient and intensity differences as indicated.

Page 5: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602 1597

brightness change will not be noticeable. Moreover, the seamshould not introduce inconsistent scene elements (i.e., by cuttingan object in half). These criteria can be largely satisfied by findingthe minimum-weight path through the combined difference im-age. As dynamic objects will have high weighted pixels, inconsis-tent scene elements will be avoided. Similarly, small brightnesschanges will be favoured over large ones, making a less notice-able seam.

Fig. 9. Stitched photos. Note that (b), (d) and (e) use optimal seam selection to ob

Table 1Panorama programs.

Program Automatic stitching

AutoStitch [6] YesPanorama Makera Manual image orderingPanorama Factoryb Seven manual point corPanavuec Single manual point cor

a http://www.arcsoft.com/products/panoramamaker/.b http://www.panoramafactory.com/.c http://www.panavue.com/.

Finding the optimal seam is simply a matter of calculating theminimum-weight path in the difference image.

Path ¼ argminij

X

i;j

Dij

The path is found using Dijkstra’s algorithm [10], modified for usein Matlab. A matrix (min_cost) keeps track of the weight of theminimum cost path from a starting pixel to every other pixel.

tain similar results, whereas (a) and (c) do not, resulting in a translucent girl.

Seam selection

Norequired Yesrespondences required Norespondence required Yes

Page 6: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

1598 A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602

At each iteration, every pixel is tested against each of its neigh-bors to see if its neighbor’s cost plus its own weight producesa smaller value than its current cost. If so, the value of that pixel

Fig. 10. Stitched pair of photos of a difficult indoor scene. One is taken with a flash anAutoStitch has trouble blending the girl in the two photos. Panorama Maker shows a cleaPanavue did not properly transform the photos and the result is poor. The present impl

Fig. 11. Stitched pair of photos of a difficult lab scene. AutoStitch was unable to matchpresent implementation handles the moving person properly.

is updated. This continues until no updates are made. Only thepixels that recently changed and their neighbors are checked ineach loop.

d the other without. Panorama Factory was unable to match the photos together.r seam due to the difference in lighting and cuts through a section of the girl’s head.ementation stitches the images together nicely.

the photos together. The other applications have a poor choice of seam. Only the

Page 7: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602 1599

Starting at the potential end point with the lowest cost, the pathis then retraced back to a starting point using the difference matrixand the weights matrix.

The optimal seam for the input images is shown in Fig. 8.

3.4. Blending

Seam selection determines the borderline between images to bemerged, but the pixels on either side of the seam will usually stillexhibit some degree of inconsistency. To minimize this saliency,we can post-process the local image content as the images areblended. The blending is done using a multi-resolution spline [8].The multi-resolution technique was chosen as it preserves highfrequency edges, which avoids ghosting and compliments the opti-mal seam selection well. The spline is outlined below.

Fig. 12. Stitched photos with significant parallax. Images were cropped. Th

1. Create a mask image using the optimal seam. All pixels on oneside of the seam are set to white and on the other side to black.

2. Run bandpass filters on both of the source images and the maskimage to obtain a fourier decomposition.

3. Blend each band of image A with the corresponding band ofimage B using the corresponding mask for that frequency band.Lower frequencies have grayscale pixels in the mask and willhave a weighted average based on the level of the gray. Thisallows the low frequency components to blend more smoothlythan the high frequency components.

4. Finally, sum the images in each frequency band to create thefinal blended image.

The number of frequency bands to use is chosen based on thesize of the overlapping region. As each additional band doubles

e other applications were unable to find a match between the photos.

Page 8: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

1600 A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602

the size of the blended region, the number of bands is chosen tomaximize the amount of the overlapping region used in blending,as more bands will create a smoother blend.

Blending the images in this manner allows low componentinformation to be dispersed and blended over a large radius, mak-ing a smooth seam even in the presence of exposure differences.This technique complements the optimal seam selection as theseam finds areas which have nearly identical high frequency ele-ments, but potentially varying low frequency elements.

The result of the process leading to a blended mosaic of the twosample images can be seen in Fig. 9e.

Fig. 13. Three photos stitched together.

3.5. Adding more images

Our approach computes a mosaic from an ensemble of images byperforming successive pair-wise combinations and gradually build-ing up a composite image. The images are first selected by the pairthat has the best match based on the ratio of RANSAC inliers to out-liers found in the feature matching step. The third image selected toblend onto the mosaic is the one having the best inlier to outlier ra-tio to either of the two images already in the mosaic. Additionalimages are stitched to the mosaic using this selection criteria. Athird image stitched to the previous result can be seen in Fig. 13.

The photos have all been cropped.

Page 9: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602 1601

4. Results

Our algorithm was compared with four other available pano-rama applications listed in Table 1. The results can be seen inthe following figures. These examples were chosen to illustratesome of the real world situations for which our algorithm wasdesigned.

� Fig. 9: This example is a simple case. The two source imagesfrom Section 3 were provided as input. There is substantial over-lap between the source images and dynamic elements take uponly about 20% of the region of overlap. All three algorithmswhich use optimal seam selection perform well.

� Fig. 10: This example illustrates exposure differences. Themotion in the scene also projects to a larger portion of the over-lap region between the images. One of the input images wasacquired using a flash and the other was taken only with naturalillumination. This is a much more difficult problem and only thepresent method produces an inconspicuous seam. To obtain asatisfactory result, exposure correction as well as optimal seamselection must be used. Not all seam selection methods willwork.

� Fig. 11: This example illustrates a scene with a largedynamic object. Panorama Maker, Panorama Factory andPanavue all make a noticeable cut through the subject’sbody. The present application chooses the pose from the leftimage rather than the right and as a consequence, makes aseamless stitch.

� Fig. 12: This example illustrates how the algorithm handlesnoticeable parallax. Most of the programs were unable to evenregister the images. The present algorithm finds a nice seam toblend the images together. This seam was very dependent onhow the edges of the difference image were inflated. The reasonsfor this are unknown.

� Fig. 13: This example shows stitching with more than twoimages and to show a scenario where an optimal seam doesnot exist. In this image there is a car which is partially in theframe, making it impossible to create a ‘perfect’ stitch as thereis no seam which either includes or excludes the car.

To run the algorithm on the images shown in Fig. 2 took 56 s.The images were each 300 � 400 pixels. Of this time, 16 s wasspent finding the optimal seam. The tests were performed on a In-tel Pentium 4 CPU @ 3.20 GHz.

5. Discussion

A technique has been presented to automatically stitch andblend together images with dynamic elements. It partially solvesthe problems of moving objects, small parallax, illuminationchanges and lens distortion. The algorithm was compared withpre-existing solutions from both the commercial field and the re-search literature, and found to produce comparable or superior re-sults to all of them.

The algorithm struggles with objects for which the entire objectcan neither be included or excluded. One method of dealing withthis would be to attempt to identify when the optimal seam cutsthrough an object. If so, a more severe cross-fade blend could beused to create a smoother transition.

The algorithm handles small parallax artifacts fairly easily, andsometimes works well with large parallax. Although to perfectlyhandle this problem full 3D recovery and registration is required,it may be possible to improve upon the algorithm’s handling ofthe problem. One problem which seemed to occur was that inthe presence of parallax-induced inconsistencies, where there is

no seam with perfect overlap in the image, the algorithm oftenchooses an optimal seam along the border of one of the images.This was discouraged by introducing an ad hoc penalty term alongthe border pixels. A more principled approach to this problemwould be attractive.

With suitable optimization, the mosaic synthesis process can beaccomplished very rapidly. For example, some techniques can ap-proach real-time speeds [7], albeit without operations such asthe optimal seam selection we employ. While our work does notcurrently addresses practical deployment issues, several interest-ing optimizations may be possible.

The choice of the formula to weight the gradient and intensitydifference images was motivated based on empirical observations,but more experimentation is needed to determine which factorsshould contribute and how.

References

[1] Y. Afek, A. Brand, Mosaicking of orthorectified aerial images, PhotogrammetricEngineering and Remote Sensing 64 (2) (1998) 115–125.

[2] A. Agarwala, Efficient gradient-domain compositing using quadtrees, in:International Conference on Computer Graphics and Interactive Techniques,2007.

[3] A. Agarwala, M. Dontcheva, M. Agrawala, S. Drucker, A. Colburn, B. Curless, D.Salesin, M. Cohen, Interactive digital photomontage, in: InternationalConference on Computer Graphics and Interactive Techniques, 2004, pp.294–302.

[4] P.M.Q. Aguiar, Unsupervised simultaneous registration and exposurecorrection. in: 2006 IEEE International Conference on Image Processing,2006, pp. 361–364.

[5] D.I. Barnea, H.F. Silverman, A class of algorithms for fast digital imageregistration, IEEE Transactions on Computers 21 (2) (1972) 179–186.

[6] M. Brown, D.G. Lowe, Recognising Panoramas, in: Proceedings of ICCV, 2003,pp. 1218–1225.

[7] M. Brown, D.G. Lowe, Automatic panoramic image stitching using invariantfeatures, International Journal of Computer Vision 74 (1) (2007) 59–73.

[8] P.J. Burt, E.H. Adelson, A multiresolution spline with application to imagemosaics, ACM Transactions on Graphics 2 (4) (1983) 217–236.

[9] J. Davis, Mosaics of scenes with moving objects, in: Computer Vision andPattern Recognition, 1998. Proceedings 1998 IEEE Computer SocietyConference on, 1998, pp. 354–360.

[10] E.W. Dijkstra, A note on two problems in connection with graphs, NumerischeMathematik 1 (1) (1959) 269–271.

[11] A.A. Efros, W.T. Freeman, Image quilting for texture synthesis and transfer, in:Proceedings of the 28th Annual Conference on Computer Graphics andInteractive Techniques, 2001, pp. 341–346.

[12] M.A. Fischler, R.C. Bolles, Random sample consensus: a paradigm for modelfitting with applications to image analysis and automated cartography,Communications of the ACM 24 (6) (1981) 381–395.

[13] R.I. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, seconded., Cambridge University Press, 2004, ISBN: 0521540518.

[14] M. Irani, P. Anandan, J. Bergen, R. Kumar, S. Hsu, Efficient representations ofvideo sequences and their applications, Signal Processing: ImageCommunication 8 (4) (1996) 327–351.

[15] A. Levin, A. Zomet, S. Peleg, Y. Weiss, Seamless image stitching in the gradientdomain, Lecture Notes in Computer Science (2004) 377–389.

[16] J. Long, D. Mould, Improved image quilting, in: Proceedings of GraphicsInterface 2007, 2007, pp. 257–264.

[17] D.G. Lowe, Distinctive image features from scale-invariant keypoints,International Journal of Computer Vision 60 (2) (2004) 91–110.

[18] B.D. Lucas, T. Kanade, An iterative image registration technique with anapplication to stereo vision, in: Proceeding of the 1981 DARPA ImageUnderstanding Workshop, 1981, pp. 121–130..

[19] K. Mikolajczyk, C. Schmid, A performance evaluation of local descriptors, IEEETransactions on Pattern Analysis and Machine Intelligence (2005) 1615–1630.

[20] D.L. Milgram, Computer methods for creating photomosaics, IEEE Transactionson Computers 100 (24) (1975) 1113–1119.

[21] S. Peleg, Elimination of seams from photomosaics, in: Conference on PatternRecognition and Image Processing, Dallas, TX, 1981, pp. 426–429.

[22] T. Porter, T. Duff, Compositing digital images, in: Proceedings of the 11thAnnual Conference on Computer Graphics and Interactive Techniques, 1984,pp. 253–259.

[23] H.Y. Shum, R. Szeliski, Systems and experiment paper: construction ofpanoramic image mosaics with global and local alignment, InternationalJournal of Computer Vision 36 (2) (2000) 101–130.

[24] P. Soille, Morphological image compositing, IEEE Transactions on PatternAnalysis and Machine Intelligence (2006) 673–683.

[25] R. Szeliski, Image alignment and stitching: a tutorial, Foundations and Trends�

in Computer Graphics and Vision 2 (1) (2006) 1–105.

Page 10: Image and Vision Computing - pdfs. · PDF fileImage stitching Mosaic Blending Computer vision ... sure parameters using a linear model of bias and gain. ... intelligent manner and

1602 A. Mills, G. Dudek / Image and Vision Computing 27 (2009) 1593–1602

[26] M. Uyttendaele, A. Eden, R. Skeliski, Eliminating ghosting and exposureartifacts in image mosaics, in: Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (CVPR), vol. 2, 2001.

[27] D.N. Wood, A. Finkelstein, J.F. Hughes, C.E. Thayer, D.H. Salesin,Multiperspective panoramas for cell animation, in: Proceedings of the 24th

Annual Conference on Computer Graphics and Interactive Techniques, 1997,pp. 243–250.

[28] Y. Xiong, K. Turkowski, Registration, calibration and blending in creating highquality panoramas, in: Fourth IEEE Workshop on Applications of ComputerVision (WACV), 1998, pp. 69–74.