Shift-Map Image Shift-Map Image EditingEditing

Yael Pritch, Eitam Kav-Venaki, Shmuel Peleg

Computer Science and EngineeringThe Hebrew University of Jerusalem,

IsraelICCV 2009

Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph

LabelingShift-Map ApplicationConcluding Remarks

Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph

LabelingShift-Map ApplicationConcluding Remarks

IntroductionIntroductionGeometric image rearrangement

is becoming more popular◦Image resizing (a.k.a. retargeting)◦Object rearrangement and removal

Early methods manipulation mostly crop and scale◦For image resizing, examining image

content and removing “less important” regions

IntroductionIntroductionSeam carving [2, 13]Continuous image warping [19,

16]Shift-map editing

◦Avoids scaling and mostly remove or shift image regions

(a) Original image(b) Video-retargeting [19](c) Optimized scale-and-stretch [16](d) Improved Seam Carving[13](e) Our shift-map editing

(a) Original image(b) Our shift-map editing(c) Video-retargeting [19](d) Optimized scale-and-stretch [16](e) Improved Seam Carving[13]

Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph

LabelingShift-Map ApplicationConcluding Remarks

Image Editing as Graph Image Editing as Graph LabelingLabeling

Shift-map◦The relative shift of every pixel in the

output image from its source in an input image

◦Represents the selected label for each output pixel

Two terms are used in computing the optimal shift-map◦Data term◦Smoothness term

Image Editing as Graph Image Editing as Graph LabelingLabeling

Input image I(x, y)Output image R(u, v)The relationship between input

image and output image is defined by◦Shift-map M(u, v) = ( , )

R(u, v) = I(x + , y + )Each output pixel can be labeled

by a shift ( , )

xt yt

xt yt

xt yt

Image Editing as Graph Image Editing as Graph LabelingLabeling

The optimal shift-map M minimizes the cost function :

◦ : data term◦ : smoothness term◦N : neighboring pixels◦ = 1

dE

sE

Single pixel data termSingle pixel data termPixel rearrangement

Pixel saliency and removal

◦S : saliency map, very high for pixels to be removed, very low for pixels not to be removed

Smoothness term for Smoothness term for pixels pairpixels pair

The smoothness term represents discontinuities added to the output image by discontinuities in the shift-map

Two neighboring location and in the output image R if

The smoothness term account color difference and gradient difference

11,vu 22 ,vu ),(, 2211 vuMvuM

))(),(( qMpMEs

Smoothness term for Smoothness term for pixels pairpixels pair

◦ : four unit vectors - four spatial neighbors

◦Color differences are Euclidean distances in RGB

◦ and are the magnitude of the image gradients at these locstion

◦ = 2

I R

ie

Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph

LabelingShift-Map ApplicationConcluding Remarks

Hierarchical Solution for Hierarchical Solution for Graph LabelingGraph Labeling

Finding the optimal graph labeling, the number of possible labels is the number of pixels in the input image

Use heuristic hierarchical approach reduces the memory and computational ◦First solved in a coarse resolution◦Higher resolution level

Hierarchical Solution for Hierarchical Solution for Graph LabelingGraph Labeling

Example : 4th pyramid level◦The number of pixels and number of

labels are reduce by a factor of 64

Coarse level

Coarse level

Input image Output image

64x64

32x32

16x16

4x4

Coarse shift-map

Nearest neighbor interpolation

Hierarchical Solution for Hierarchical Solution for Graph LabelingGraph Labeling

Use three to five pyramid levelsThe coarsest level contains up to

100 x 100 pixels

Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph

LabelingShift-Map ApplicationConcluding Remarks

Shift-Map ApplicationShift-Map ApplicationImage retargetingImage rearrangementInpainting Image composition

Image retargetingImage retargetingLabel order constraint

◦The shift-map will retain the spatial order

◦In the case of reducing width and , ,

◦In the case of increasing width and , ,

),(),( yx ttvuM ),(),1( ''yx ttvuM xx tt ' 0xt

),(),( yx ttvuM ),(),1( ''yx ttvuM xx tt ' 0xt

Image retargetingImage retargetingControlling object removal

◦It is possible to control the size and number of removed objects by performing several steps of resizing

◦Also possible to control object removal by marking objects as salient

◦The number of steps becomes the number of removed columns

(a)Original image(b) Resizing in single

step(c) Six smaller resizing

steps(d) Ten smaller resizing

steps

Shift-map retargeting :(a) Original image(b)(c)(e) No saliency(d) Child was marked salient

Original image [13] [19] [16] shift-map

Image rearrangementImage rearrangementMoving an object to a new image

locationDeleting part of the imageSpecified in two parts using the

data term◦Force pixels to appear in a new

location using Eq. 2◦Marks these pixels for removal from

their original location using Eq. 3

Example – 1 : ◦move the person and a part of the

temple to the right, and keep the tourists at their original location

Example – 2 :◦Kid on the left should move to the

center, baby should move to the left, kid on the right should remain in place

InpaintingInpaintingUnwanted pixels are given an

infinitely high data term as described in Eq. 3

Maps pixels inside the hole to other locations in the input image

InpaintingInpaintingA good complition with no user

intervention

Image compositionImage compositionIn the shift-map framework the

input can consist of either a single image, or of a set of images◦ , is the index of

the input imageTolerate misalignments between

the input images

),,(),( indyx tttvuM indt

Outline Outline IntroductionImage Editing as Graph LabelingHierarchical Solution for Graph

LabelingShift-Map ApplicationConcluding Remarks

Concluding RemarksConcluding RemarksShift-maps are proposed as a new

framework to describe various geometric rearrangement problems

Images generated by the shift map are natural looking

Minimal and intuitive user interaction

Distortions that may be introduced by stitching are minimized

Large regions can be synthesized

Thank you!!!Thank you!!!