Kabul2011 Texture Metamorphosis

8/4/2019 Kabul2011 Texture Metamorphosis

1/12

Volume 0 (1981 ), Number 0 pp. 112 COMPUTER GRAPHICS forum

An Optimal Control Approach

for Texture Metamorphosis

Ilknur Kabul1, Julian Rosenman 1,2, Stephen M. Pizer1 and Marc Niethammer1

1 Department of Computer Science, UNC Chapel Hill, USA2 Department of Radiation Oncology, UNC Chapel Hill, USA

Abstract

In this paper, we introduce a new texture metamorphosis approach for interpolating texture samples

from a source texture into a target texture. We use a new energy optimization scheme derived fromoptimal control principles which exploits the structure of the metamorphosis optimality conditions. Ourapproach considers the change in pixel position and pixel appearance in a single framework. In contrastto previous techniques that compute a global warping based on feature masks of textures, our approachallows to transform one texture into another by considering both intensity values and structural featuresof textures simultaneously. We demonstrate the usefulness of our approach for different textures, suchas stochastic, semi-structural and regular textures, with different levels of complexities. Our methodproduces visually appealing transformation sequences with no user interaction.

Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three-Dimensional Graphics and RealismColor, shading, shadowing, and texture

1. Introduction

While image registration only deals with the align-ment of a source to a target image, image metamor-phosis also considers changes in image appearance al-lowing to compute transitions from a source to a tar-get image. Consequentially, image metamorphosis hasbeen of increasing importance in computer graphics aswell as in medical imaging. It has been used to morphbetween faces, to mix multiple images, and to accom-modate for appearance changes in image registrationin general.

To obtain an image metamorphosis requires the com-putation of point correspondences over time (a time-dependent warp field) as well as an optimal change

of pixel intensity to allow for an exact match to thetarget image. This joint optimization problem has forexample been studied in [TY05,Hol09] and is a naturalextension of fluid-flow registration methods which esti-mate time-dependent velocity fields to smoothly trans-form one image into another [BMTY05]. Figure 1 il-lustrates the difference between registration and meta-morphosis. In contrast to key-framing, for cartoon-

like images the image metamorphosis approach does

not take into account kinematics, but computes thespace deformation using image information (and pos-sible feature channels) only.

Texture metamorphosis is a special case of imagemetamorphosis which is used to design new texturesby interpolating between the input textures or to visu-alize the change of material on the objects by creatinga smooth transition between two textures. Structurefeatures and appearance of the interpolated texturesshould perceptually be between input texture-pairs.Creating a meaningful and smooth transformation isa difficult problem. Transformation of the intensityvalues as well as the feature masks, such as edges and

ridges, should have an effect in the computation of thewarping in order to split or merge structural features.

In this paper, we present a new texture metamor-phosis approach considering structural and appear-ance transitions of textures in a single framework.Our method uses an optimal control framework (asproposed in [HZN09] for the image registration prob-lem) and proposes a new numerical solution method

c 2011 The Author(s)

Journal compilation c 2011 The Eurographics Association and

Blackwell Publishing Ltd. Published by Blackwell Publishing, 9600

Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street,

Malden, MA 02148, USA.


2/12

Kabul, Rosenman, Pizer, & Niethammer / An Optimal Control Approach for Texture Metamorphosis

Figure 1: Comparison between registration and meta-morphosism.

which solves the metamorphosis problem as an ini-tial value problem (and can therefore exploit a nu-merical solution strategy from a widely used fluidregistration approach [BMTY05]). It is based onthe large-displacement-diffeomorphic metric mapping

(LDDMM) formulation of fluid registration [BMTY05,HZN09].

Our approach results in an intuitive solution methodfor texture metamorphosis and has a number of advan-tages compared to other approaches for texture meta-morphosis:

1. It is designed to interpolate textures while consider-ing appearance and features in a single framework.This removes the need for an additional blendingor synthesis scheme.

2. It can easily accommodate different image informa-tion, such as color or structural feature masks.

3. It can easily accommodate different measures ofimage/feature similarity, such as simple sum ofsquared differences, correlations, or mutual infor-mation.

4. It is designed to be symmetric with respect to thetwo textures.

5. It is robust and automatic. It neither requires userinput for feature correspondances nor the selectionof transition functions.

6. It interpolates structural textures as well asstochastic and natural textures.

The remainder of the paper is organized as follows:Section 2 discusses related work on texture metamor-phosis. Section 3 presents the overview of our method.Section 4 illustrates our optimal control based texturemetamorphosis approach. How to add color informa-tion is explained in Section 5. Section 6 details our tex-ture metamorphosis approach for structural textures.Section 7 presents results.

2. Background

Several approaches have been proposed to find thetransformation from one image to another. Bar etal. [BJEYLW01] proposed a semi-morphing approachin which 50/50 blends of textures are created by

performing texture analysis using wavelets. The firstapproach for morphing structural textures was pre-sented in [LLSY02] using a pattern-based approachin which the user specifies landmarks and their cor-respondances for two textures. The landmarks areused to create a warp field and cross-dissolving accom-modates texture appearance changes. This approachassumes that texture images are composed of simi-lar texture patterns. In addition, it requires a largeamount of user interaction. Later, in [LiW03] an al-gorithm for synthesizing texture mixtures from differ-ent input textures is presented. In this method, weightimages are used to control how each source texture ef-fects the result texture.

Zhang et al. [ZZV03] synthesized progressively-variant textures by using a texton mask as the featureimage for input textures. In their method, first a mixedfeature mask is generated by blending and binary-thresholding. Then this mixed feature mask is used tosynthesize the mixed output texture. The main disad-vantage of this approach is that it does not guarantee asmooth transformation from one image to another. Inaddition, an arbitrary number of images between twoinput images cannot be generated. Liu et al. [ LLH04]propose a system to design novel textures by analyz-ing and manipulating textures. Their method allowsthe user to manipulate texture features which can be

used for alignment in morphing. User defined featuremaps are also used in [MZD05] to produce warps be-tween textures. They also propose a histogram match-ing technique to avoid blurring of features. Their ap-proach is only applicable to textures that are similar toeach other. A level set advection method for morphingbetween texture features was proposed in [RLW09].This method first finds the warped texture features be-tween two texture feature images. Then a constrainedtexture synthesis approach is applied to generate tex-ture with a given feature mask. Recently, a new patchbased approach was proposed in [RSK10]. In this ap-proach, initially individual neighborhoods of the inputtextures are locally warped and blended and then they

are used to create new textures by an optimization al-gorithm. The disadvantage of this approach is that itcannot handle materials with large differences in fea-ture scales. In addition, since the warping is done us-ing only feature information, it cannot handle texturesthat do not have clear, meaningful features.

The metamorphosis formulation that our approachbuilds on has been studied for example in [TY05,



Blackwell Publishing Ltd.


3/12


Hol09] and numerical solution methods have been pro-posed in [MY01] and [GY05]. This line of work hasmostly been concerned with morphing gray-valued orcolor images and has so far not dealt with the problemof texture morphing.

The use of optimal control formulations for im-age time-series have been proposed in [BIK02, CL10,NHZ09]. In contrast to metamorphosis, these ap-proaches address image deformations only, but notappearance changes. Furthermore, in contrast toour metamorphosis formulation and [NHZ09, HZN09],[BIK02,CL10] solve the associated transport equationfor the image directly, which requires advanced nu-merical methods to avoid image blurring.

3. Overview

Given two texture images I0 and I1, we want to createinterpolated images I(t). Here, t corresponds to timet [0, 1] under the constraint that I(1)= I1 and I(0)=

I0, which enforces that the initial and the final textureimages are matched exactly under metamorphosis.

Since there are infinitely many possible ways to morphimage I0 into I1, the solution space needs to be con-strained in a meaningful way. In metamorphosis thisis accomplished by formulating an optimization prob-lem which seeks to determine a time-dependent veloc-ity field v and a time-dependent source term q whichare appropriately regularized.

In an optimal control setting the metamorphosis opti-mization problem is equivalent to minimizing the fol-lowing energy E under a dynamic constraint for theimage change:

E(v, q) =

10

v2L +q2Q dt, (1)

subject to It + (DI)v = q, I(0) = I0, I(1) = I1.

Here, the control variables v (the time and space de-pendent velocity field) controls the image deformation(flow of image over time), q controls the change in ap-pearance over time, and D denotes the Jacobian. Land Q denote norms to penalize v and q. The defor-mation map can be computed from v.

The following constraints are imposed onto the opti-mization problem:

1. Image Flow: This constraint is simply a transportequation with a source term q and controls the flowof the image from I0 to I(t). Intuitively it can beconsidered as a constraint which imposes that im-age intensity along a streamline of the velocity fieldv changes over time only through q. For q = 0 in-tensities will be constant along such a streamline(characteristic).

2. Initial and final constraints: To allow for texturemorphing between two texture exemplars, the ini-tial and the final condition for I are fixed. This isthe main difference to standard image registrationwhere an inexact match of the warped source imageto the target image is admissible. This requires de-

termining the time and spatially dependent q suchthat the exact match is achieved under the imageflow constraint.

Solving the optimization problem in equation 1 is chal-lenging. We make use of an adjoint solution method asproposed in [HZN09] for image registration. Details ofthe solution method and its relation to the correspond-ing image-to-image registration approach are given inthe next section.

4. Texture Metamorphosis

Image registration involves finding a transformationwhich maps one image to another image as well as

possible. In contrast to image metamorphosis, imageregistration is only concerned with determining thismapping and does not consider appearance changes.Therefore perfect image matching is in most cases notachievable in image registration due to image noise,structural differences or appearance differences in theimages.

In fluid flow registration,the deformation is achievedby flowing the image I0 with respect to a time-dependent velocity field v. The fluid flow registrationproblem is highly complex, because the full spatio-temporal velocity field needs to be estimated. To makethis problem computable, non-smoothness of the ve-locity field is penalized, to obtain spatially coher-ent registration solutions [BMTY05, HZN09]. In fact,by choosing an appropriate norm to penalize non-smoothness diffeomorphic image transformations canbe assured.

The energy to be minimized in large-displacementdiffeomorphic metric mapping (LDDMM) [BMTY05,HZN09] fluid flow registration is

E(v) =

v2L dt +

1

2I(1)I1

2, (2)

subject to It + (DI)v = 0, I(0) = I0

where v2L = Lv,Lv and L is a differential operatorto encourage smoothness of the velocity field, typi-

cally chosen of the form L = 2 + . Here en-courages smoothness of the vector field. For instance,for large strong deformations are discouraged andthe interpolations looks like blending; controls howmuch the overall travel distance of a particle is countedin the energy. Intuitively, a large will favor smalloverall displacements over an accurate image match. is a scalar which controls the tradeoff between image





4/12


match and transformation smoothness. Note that theonly difference to the metamorphosis problem in equa-tion 1 is that the final constraint I(1) = I1 has beenreplaced by an inexact matching term I(1) I1

2.See [HZN09] for details on how to solve such an opti-mization problem.

To allow for exact matching, the metamorphosis for-mulation 1 adds a control variable q to the transportequation which controls the image intensity changeand is penalized according to q2Q, where q

2Q =

Qq,Qq and Q is a chosen differential operator. (Notethat for an easy numerical solution the operator is cho-sen as Q = id, the identity.) In Equation 1, the weight-ing parameters of the source term q and the velocityterm v are set to the same value. Changing the weightsof these terms would produce different results. For ex-ample, increasing the weight for the source term anddecreasing the weight for the velocity term would shiftthe balance towards stronger deformation and less ag-

gressive blending.To solve this constrained optimization problem, weconvert it to its unconstrained form by adding the dy-namic constraint through a time and space dependentLagrangian multiplier . We add the final state condi-tion through the Lagrangian multiplier and obtainthe unconstrained energy:

E(v,q,,I,) =

10

v2L +q2Q

+, It + (DI)vq dt + , I(1)I1

with respect to v(x, t), q(x, t), I and the Lagrangianmultipliers (x, t), (x).

For a candidate minimizer of the unconstrained en-ergy, its variation with respect to v,q,I, , and needto vanish (Details of how the differential equations aresolved is presented in the Appendix). Computing thefirst variation and rearranging terms yields

E(v,q,,I; dv,dq,d,dI) =

10

2LLv + (DI)T, dv

+2QQq,dq+ It + (DI)vq,d

+tdiv(v), dI dt + ,dI|10

+d,I(1)I1+ ,dI(1).

Since E needs to vanish for any dv, dq, dI, d, and d,fulfilling the boundary conditions (i.e. zero boundaryconditions for v, I(0) = I0, I(1) = I1) at optimalitythe following conditions need to hold:

1. Initial Condition: Sets the starting point for theimages at time t = 0.

I(0) = I0

2. State Equation (Transport Equation) : Flows theimage forward to time t = 1 using the velocity fieldv. The source term in the transport equation sim-ply changes image appearance and thus makes theappearance and disappearance of structures in animage possible.

It + (DI)v = q

3. Final Conditions : Sets the final image and providesa final conditon for the adjoint.

(1) = 2q(1), I(1) = I1 (3)

4. Adjoint Equation (Scalar Conservation Law) : Gov-erns the dynamics of the adjoint (Lagrangian mut-liplier ). Since the final condition (1) is givenby 3 this equation can be solved backward in time.Note that the adjoint corresponds to the concept ofmoments in physics [MTY06] and the scalar conser-vation law for the adjoint consequentially resemblesthe physical law of conservation of momentum.

tdiv(v) = 0

5. Compatibility Condition:

2LLv + (DI)T = 0, 2q = 0

The first equation is used to compute the gra-dient of E with respect to v on the intervalt = [0, 1]. From E(v,q,,I; dv,dq,d,dI) =102LLv + (DI)T, dvdt, the gradient

L2v E(v) = 2LLv + (DI)T with respect to

the L2 norm or Vv E(v) = 2v + (L

L)1((DI)T)

in the V-norm (induced by the operator L) isobtained.

The second equation illustrates that q is directlyrelated to the adjoint (subject to a potentialsmoothing operator Q). To simplify the problem,we assume that the appearance control is penal-ized by an L2 norm which makes Q = id and thesecond compatibility condition becomes 2q = 0.

The optimality conditions reveal an interesting struc-ture: the only difference to the image-to-image reg-istration [HZN09] case are the source term q in thetransport equation (the forward model), the addi-tional compatibility condition on q, as well as the final

condition on I.

4.1. Adjoint Solution Method

An analytic solution can in general not be ob-tained from the optimality conditions for metamor-phosis. Numerical solution methods have been pro-posed in [MY01] and [GY05]. In both cases, the meta-





5/12


morphosis problem is cast as a boundary value prob-lem, subject to the fixed source and target images andthe solution proceeds by alternating gradient solutionschemes for the image values (implicitly accounting forthe image source terms, q, in the transport equation)and for the velocity field.

Here, by exploiting the relation between and q, wesidestep an iterative update of the image I and onlyperform gradient descent with respect to the velocityfield v. This approach results in an initial value prob-lem which allows using a similar map-based solutionas in [BMTY05], reveals the simple structure of theunderlying optimality conditions and allows to easilyintegrate alternative image-matching terms (as theyonly affect the estimates for the velocity field throughthe final condition for the adjoint; see [HCF02] forcomputations of these final conditions). While the useof maps is not strictly necessary for the solution of theoptimization problem [HZN09] it is beneficial to avoid

smoothing due to numerical dissipation effects.

Algorithm 1 Texture metamorphosis

Require: I0, I1, LEnsure: v

Assume an intial flow field v is givenrepeat

Flow image forward usingIot + (DI

o)v = 0, I(0) = I0Find q(1) according to Eq. 8Find adjoint (1) from q(1) using q = 2Flow adjoint (t) backward using Eq. 6Compute q(t) from the solution of the adjoint us-ing q = 2

Compute the image I(t) according to Eq. 7Compute the gradient for v from the compatibil-ity conditionsPerform a gradient descent step using vE

until Convergence

Specifically, given an estimate of the velocity field v,we compute the deformation maps t and

rt from

t = 0 to t = 1 using

t + (D)v = 0, (0) = id, (4)

rt (Dr)v = 0, r(1) = id (5)

where id denotes the identity map, D denotes the Ja-

cobian, and r

t maps to t = 1. This allows to replacethe direct solution for and I by solutions of transportequations on the maps and r. The advantage ofthe map-based approach is that the maps are expectedto be smooth (since the velocity fields are regularizedaccording to L) and are therefore easier to propagatenumerically.

To compute at each time step according to the ad-

Figure 2: Optimal Control Solution Framework forcomputing I(t).

Figure 3: Optimal Control Solution Framework forcomputingvE.

joint equation, (1) is transported backwards in timeusing r(t). In order to conserve the values over time,the transformation is multiplied by |Dr(t)|, whichcorresponds to the amount of deformation the maphas undergone:

(t) = |Dr(t)|(1)r(t). (6)

Using these maps to compute the solutions I(t) isslightly more complicated than in the standard im-age registration problem [BMTY05,HZN09] due to thesource term q, which needs to be integrated along thecharacteristics (streamlines) of the velocity field to af-fect the overall intensity value. This can be expressedby adding the change of intensity to the transformedimage I0 (t) as

I(t) = I0 (t) +

t0

q()t, d, (7)

where t, = (t)1()

which simply sums up all the q contributions over timein the current reference coordinate system at time t.

Our proposed solution strategy builds on the obser-vation that given a velocity field v we can compute

the final condition (1) related to the source term qwhich results in I(1)= I1 exactly. This is possible sincewe know from the optimality conditions that q = 2 .Specifically, we observe that

q(1) =I1I

o(1)10

1|D1,t|

dt(8)

pointwise (The details of this equation are explained in





6/12


Figure 4: The texture sequences along with the defor-mation fields obtained in different steps of iteration.

Figure 5: Metamorphosis from one texture to anotherusing image intensities. Top row shows linear interpo-lation and bottom row shows our results.

the Appendix.). Given q(1) we can compute (1) andin turn q(t). Here, Io(1) denotes the solution to thesource-less transport equation Iot +(DI

o)v = 0; I(0) =I0 using the given velocity field v.

The so-called adjoint solution method proceeds ac-cording to the Algorithm 1 which is illustrated inFigure 2 and Figure 3. In effect, given an estimateof v one (i) computes the solution to the source-lesstransport equation, (ii) uses it to compute the finalcondition for q, (iii) which then allows the computa-tion of(t), q(t), and I(t), (iv) from which the gradientv(t)E can be computed. After taking a gradient de-scent step with respect to the velocity field, these stepsare repeated to convergence. Figure 4 shows the in-termediate results along with the deformation fields ineach step.

Figure 5 illustrates our metamorphosis results. Eventhough there is no control in the structural features

of the textures, our approach can split and merge thepatterns by only using the q term.

4.2. Combining Forward and Backward

Transition in Metamorphosis

According to the constraint I(1) = I1, the morphedimage I(1) should always match the target image I1.

However, due to numerical inaccuracies, the constraintmay not be satisfied exactly. In order to avoid thisproblem, we apply the solution method in the forwarddirection and the backward direction, which allows usto make the problem symmetric and to obtain exactmatching by construction. Specifically, we use two ap-

pearance control terms qF and qB and weight themlinearly. Here, qF controls the change in appearancefrom I0 to I1 and qB controls the change in appear-ance from I1 to I0. The modified energy function is

E(v, q) =

10

v2V + (1 t)qF2Q + tqB

2Q dt, (9)

subject to IF,t + (DIF)v = qF,

IB,t + (DIB)v = qB ,

IF(0) = IB(0) = I0, IF(1) = IB(1) = I1

where IF(t) denotes the transformed image in the for-ward direction from I0 to I1 at time step t, and IB(t)

refers to the transformed image in the backward di-rection from I1 to I0 at time step t.

The resulting optimality conditions are similar to theones obtained previously (now for two sets of equa-tions) except for the compatibility condition whichbecomes

2LLv + (1 t)F(DIF)T+ tB(DIB)

T = 0. (10)

Note that this simply amounts to solving the problemtwice (in opposite directions) while keeping only onevelocity field that needs to be estimated. The gradientfor the velocity field is then a linear combination ofthe gradients for the image solutions in the differentdirections and the image itself is recovered as

I(t) = (1 t)IF(t) + tIB(t).

In what follows we assume that all equations aresolved numerically in this way without explicitly writ-ing down the resulting forward/backward equations.

5. Metamorphosis for Color Textures

In texture metamorphosis, the traditional approach isto warp single intensity values which come either fromthe texture image or its feature mask. In our method,multiple channels can be easily integrated into the en-ergy equation by adding q terms for each channel. Theenergy equation becomes:

E(v, q) =

1

0

v2L +ni=1

qi2Q dt, (11)

s.t. Iit + (DIi)v = qi, Ii(0) = Ii0, I

i(1) = Ii1 (12)

where n is the number of channels that should be con-sidered in the optimization framework. In our exam-ples, we used the red, green and blue color channels.





7/12


Figure 6: First and second rows show the results ob-tained using the approach in [RLW09 ]. Third and

fourth rows illustrate our results.

For each channel independently, the same optimality

conditions for metamorphosis are obtained (i.e., initialand final condition, transport equation, conservationlaw). The compatibility conditon for v which is usedto compute the gradient changes becomes:

2LLv +

ni=1

i(DIi)T = 0. (13)

6. Metamorphosis for Structured Textures

Some of the textures have strong structural compo-nents, such as edges, ridges, etc. These features shouldbe considered in metamorphosis in order to obtain vi-sually pleasing results since the structural components

may undergo topological changes such as splitting andmerging. In this optimization framework, our aim is tosimultaneously morph one image into another with re-spect to appearance (intensities) while aligning theirfeature masks (without appearance change). This canbe accomplished by adding the feature mask as aninexact matching term (as in image registration) sub-ject to a transport equation withoutsource term. Theenergy equation to be minimized becomes

E(v) =

10

v2L +q2Q dt +

1

2F(1)F1

2, (14)

s.t. It + (DI)v = q, I(0) = I0, I(1) = I1,

Ft + (F I)v = 0, F(0) = F0.

This can be combined with multiple color channels formetamorphosis as desired. Note that the feature imagedoes not have a final state constraint.

Adding the inexact matching term results in optimal-ity conditions for F and its adjoint which are thesame as for image registration [HZN09]. The compat-ibility condition for v changes respectively to include

the influence of the feature image (i.e., the feature im-age influence is added). All other optimality conditionsremain the same as in Section 4:

Ft + (DF)v = 0, F(0) = F0,

tdiv(v) = 0,

2LLv + (DI)T + (DF)T = 0,

(1) =2

2(F(1)F1).(15)

See Algorithm 2 for a description of the solution steps.

Algorithm 2 Feature-based texture metamorphosis

Require: I0, I1,F0, F1, LEnsure: v

Assume an intial flow field v is givenrepeat

Flow image forward usingIot + (DI

o)v = 0, Io(0) = I0Flow feature image forward usingFt + (F I)v = 0, F(0) = F0Find q(1) according to Eq. 8Find adjoint (1) from q(1) using q = 2Flow adjoint (t) backward using Eq. 6Flow adjoint (t) backward usingtdiv(v) = 0, (1) =

22

(F(1)F1)Compute q(t) from the solution of the adjointCompute the image I(t) according to Eq. 7Compute the gradient for v from the compatibil-ity conditionsPerform a gradient descent step using the gradi-ents vE

until Convergence

In Figure 6, the comparison of our approach witha texture synthesis based interpolation approach isshown. As can be seen in this figure, the intensitychanges in the interpolated images are not smooth inthe results obtained using the approach of [RLW09].The bright regions inside the pattern are getting big-ger first and then they are getting smaller to matchwith the target image. On the other hand, in our re-sults, the bright regions move inside the patterns togenerate the regions in the target image.

7. Results

Our technique is suited for regular, semi-regular and

stochastic textures, since we perform the deformationand appearance change in a single framework. In Fig-ure 7, we illustrate the results our approach gener-ated using only image intensity information. As can beseen, our approach created textures which are sharpand detailed, even when it is hard to define struc-tures and the correspondences between them. Espe-cially, the interpolation between pink flowers to white





8/12


Figure 7: When meaningful features cannot be defined in the textures, such as natural textures, our metamorphosiscan morph between textures by only considering appearance information. For each texture pair, first column showsthe results with linear interpolation and second column illustrates the results with our approach.

Figure 8: Including feature channel in metamorphosis change the deformation (a) Linear interpolation (b) Meta-morphosis without using structural features (c) Metamorphosis using structural features.





9/12


Figure 9: Comparison of interpolation results from[RLW09] with our results. (left) Linear blend-ing (middle) Our approach (right) Results from[RLW09].

flowers created new flowers that have features presentin both input images. The interpolated textures in thisfigure can be easily used in texture synthesis to gen-erate textures on the surface of 3D models in order tovisualize the change of materials over regions.

In Figure 8, we show the effect of using featurechannels in metamorphosis. We obtained smooth andseamless transitions by propagating structural fea-tures and appearance by constraining the change inthe feature along with intensity. Our results show thatthe addition of feature channels in metamorphosis pro-

vides more control over deformation.In Figure 9 and Figure 10 , we show the compari-son of our results with those presented [RLW09]. InFigure 10, since there are no meaningful structuralpatterns in the input textures, we only used image in-formation in the energy equation. Our approach cre-ated smooth transformations from the leopard textureto the dice texture by forming the dots on the dicefrom the spots on the leopard texture. In Figure 9,we used the structural feature information along withthe image information to create interpolated textures.Since the method presented in [RLW09] interpolatestexton masks using an advection algorithm and thensynthesizes interpolated textures constrained by these

masks, the transition of the intensity values is notsmooth and perceptually pleasing. In addition, the in-terpolated textures may contain new structures whichare not available in either of the input textures. Onthe other hand, our approach deforms the structuralfeatures in the texture along with their intensity val-ues. In Figure 11, we compared our results with thosepresented in [RLW09] and [RSK10]. In this compar-

Figure 13: Top row illustrates results obtained by set-ting is 0.01, and bottom row illustrates results ob-tained by setting is 0.001

Figure 14: Failure cases

ison, the textures in the middle of the sequence aredifferent. Our approach seems to represent a nice com-promise between blending of color while at the sametime achieving a good transition in terms of texturestructure.

In Figure 12, we compare our results with the imageregistration method proposed in [HZN09]. We applyregistration from I0 to I1 and I1 to I0, and combine

the results obtained from them using linear interpo-lation. As it is shown in the figures, the results ob-tained using registration and interpolation look moreblurry. For the dots example in the first row, since thecolor channels are treated independently in registra-tion and there are no objects of corresponding colorsin the source and target images, the algorithm shrinksthe dots.

In Figure 13, we illustrate the effect of changing inthe differential operator L. Here, it can be seen that forlarge strong deformations are discouraged, and thetransformation becomes like blending. In Figure 14,some cases in which our approach fails are presented.

For the textures that have many overlapping struc-tural features our approach sometimes fails to createvalid intermediate textures.

The implementation of the proposed approach is donein Matlab and the source code is not optimized. Therun time is usually in 5-10 minutes for 64x64 tex-ture images. A GPU implementation of our approachwould run within a few seconds.





10/12


Figure 10: Comparison of interpolation results from [RLW09] with our results. Top row shows the resultsobtained using linear interpolation, middle row illustrates the results from [RLW09 ] and the bottom row showsthe results obtained using our approach.

Figure 11: Comparison of results from with our results. (a) Linear blending (b) Results from [RSK10] (c) Resultsfrom [RLW09] (d) Our approach.





11/12


Figure 12: For each initial and target texture, top row shows the results obtained using registration and linearinterpolation, and bottom rows show the results obtained using our metamorphosis approach

8. Conclusion and Future Work

In this paper, we have presented a novel optimal con-trol based texture metamorphosis approach, whichwarps one texture into another by considering the ap-pearance and feature information in a single frame-work. In contrast to other techniques that are basedon warping in texture features and texture synthe-

sis (or blending) in creating interpolated textures, ourapproach can create consistent and visually pleasinginterpolated images, even when it is not possible to de-fine the features in the texture. In future work, we willintegrate alternative features and landmarks into theenergy equation in order to better control the transfor-mation of the textures. Moreover, we only investigatedthe interpolation between two input textures. It wouldbe interesting to design a new optimization frameworkwith additional penalty terms and constraints to per-form the interpolation between a number of input tex-tures. In addition, metamorphosis between 3D solidtextures would be interesting future work.

Acknowledgement

This material is based upon work supported by theNational Science Foundation (NSF) under Grant No.EECS-0925875 and by the National Institutes ofHealth (NIH) under Grant Nos. (2 P41 EB002025-26A1, 1 R01 MH091645-01A1). Any opinions, find-ings, and conclusions or recommendations expressedin this material are those of the author(s) and do notnecessarily reflect the views of NSF or NIH.

Appendix

Optimality Conditions

For a minimizer of energy, its variation with respectto v, , I, and need to vanish. Computing


E(v + dv,I+ dI, + d) |=0

yields


10

2LLv,dv

+2q,dq+ d,It + (DI)vq+ , (dI)t

+(DdI)v + (DI)dvdq dt + d,I(1)I1+ ,dI(1).

Since10

,dItdt =

10

t, dIdt + ,dI|10,

and (by Greens theorem) , (DdI)v is equal to

div(v), dI+

dIv dS= div(v), dI,

we get


10

2LLv

+(DI)T, dv+ 2QQq,dq+ It + (DI)vq,d

+tdiv(v), dI dt + ,dI|10

+d,I(1)I1+ ,dI(1),





12/12


assuming zero boundary conditions for v and a knownI(0) = I0. Since E needs to vanish for any dv, dI, d,fulfilling the boundary conditions of the problem, theoptimality conditions are obtained.

Source Term

To be able to compute the values for q without implic-itly obtaining them through a gradient descent on I,we can make use of the fact that for the L2 norm pe-nalizer, q is directly related to the adjoint and there-fore also has to fulfill a scalar conservation law. Notethat for a transport equation without a source term,values stay constant along the characteristics (stream-lines) of the transport equation. With a source termthe source values get integrated along the characteris-tic. Hence, along a characteristic

I(t) = I(0)+

t0

q(t) dt

needs to hold In particular, for our exact matching

problem we need to have

I1I0 =

10

q(t) dt.

However, representing everything in the coordinateframe of image I1 we know that

q(t) =1

|D1,t|q(1),

since it needs to fulfill a scalar conservation law. As-suming we discretize time into n intervals spanning[0, 1] and assuming q is piecewise constant over thetime intervals

n1i=0

1|D1,t|q(1)t = I1Io(1),

which can be solved for

q(1) =I1I

o(1)n1i=0

1|D1,t|

t.

In the limit as t 0 the sum becomes an integral andwe get the desired expression (which holds pointwise)

q(1) =I1I

o(1)10

1|D1,t|

dt.

expressed in the coordinate system of image I1.

References[BIK02] Borz A., Ito K., Kunisch K.: Optimal control

formulation for determining optical flow. SIAM J. Sci.Comput. 24 (March 2002), 818847. 3

[BJEYLW01] Bar-Joseph Z., El-Yaniv R., LischinskiD., Werman M.: Texture mixing and texture moviesynthesis using statistical learning. IEEE Transactionson Visualization and Computer Graphics 7 (2001), 120135. 2

[BMTY05] Beg M., Miller M., Trouv A., YounesL.: Computing large deformation metric mappings viageodesic flows of diffeomorphisms. International Journalof Computer Vision 61, 2 (2005), 139157. 1, 2, 3, 5

[CL10] Chen K., Lorenz D. A.: Image sequence inter-polation using optimal control. CoRR abs/1008.0548(2010). 3

[GY05] Garcin L., Younes L.: Geodesic image match-ing: A wavelet based energy minimization scheme. Lec-ture notes in computer science 3757 (2005), 349. 3, 4

[HCF02] Hermosillo G., ChefdHotel C., FaugerasO.: Variational methods for multimodal image match-ing. International Journal of Computer Vision 50, 3(2002), 329343. 5

[Hol09] Holm D.: Eulers fluid equations: Optimal con-trol vs optimization. Physics Letters A 373, 47 (2009),43544359. 1, 2

[HZN09] Hart G. L., Zach C., Niethammer M.: Anoptimal control approach for deformable registration. InProceedings of the Workshop on Mathematical Methodsin Biomedical Image Analysis (MMBIA) (2009). 1, 2, 3,4, 5, 7, 9

[LiW03] LiWei2003: Texture synthesis from multiplesources. In SIGGRAPH 2003 Sketches and Applications(2003). 2

[LLH04] Liu Y., Lin W.-C., Hays J.: Near-regular tex-ture analysis and manipulation. ACM Trans. Graph. 23,3 (2004), 368376. 2

[LLSY02] Liu Z., Liu C., Shum H.-Y., Yu Y.: Pattern-based texture metamorphosis. In PG 02: Proceedingsof the 10th Pacific Conference on Computer Graphicsand Applications (Washington, DC, USA, 2002), IEEEComputer Society, p. 184. 2

[MTY06] Miller M. I., Trouve A., Younes L.:Geodesic shooting for computational anatomy. Journalof Mathematical Imaging and Vision 24 (2006), 209228. 4

[MY01] Miller M. I., Younes L.: Group actions, home-omorphisms, and matching: A general framework. In-ternational Journal of Computer Vision 41, 1/2 (2001),6184. 3, 4

[MZD05] Matusik W., Zwicker M., Durand F.: Tex-ture design using a simplicial complex of morphable tex-tures. ACM Trans. Graph. 24, 3 (2005), 787794. 2

[NHZ09] Niethammer M., Hart G. L., Zach C.: An op-timal control approach for the registration of image time-series. In Conference on Decision and Control (2009),pp. 24272434. 3

[RLW09] Ray N., Lvy B., Wang H., Turk G., runoVallet B.: Material-space texturing. Computer Graph-ics Forum (2009). 2, 7, 9, 10

[RSK10] Ruiters R., Schnabel R., Klein R.: Patch-

based texture interpolation. Computer Graphics Forum29, 4 (June 2010), 14211429. 2, 9, 10

[TY05] Trouve A., Younes L.: Metamorphoses throughlie group action. Foundations of Computational Mathe-matics (2005), 173198. 1, 2

[ZZV03] Zhang J., Zhou K., Velho L., Guo B., ShumH.-Y.: Synthesis of progressively-variant textures on ar-bitrary surfaces. ACM Trans. Graph. 22, 3 (2003), 295302. 2




Documents

Kabul2011 Texture Metamorphosis