Download pdf - NONLINEAR SEMIGROUPS AND INFINITE HORIZON …yakov/bellman.pdfNONLINEAR SEMIGROUPS AND INFINITE HORIZON OPTIMIZATION 3 The discrete time problem. Let Xbe a phase state of the model

NONLINEAR SEMIGROUPS AND

INFINITE HORIZON OPTIMIZATION

S. Yu. Yakovenko and L. A. Kontorer

§1. Introduction

1.1. Economics. In this paper we make an attempt to discuss an approach toinfinite horizon stationary optimization problems, based on the notion of (nonlinear)Bellman operators.

These problems are not merely of an academic interest, since they constantlyappear in the normative theory of growth and capital accumulation in mathemati-cal economics. Speaking rather vaguely, the normative theories describe economicdynamics as follows. There is defined (either explicitly or implicitly) a family offeasible paths, or feasible trajectories, describing a possible dynamics of a system.Usually this is done using the so called technological restrictions (e.g., balanceequations and capacity limitations are to be satisfied independently of the adoptedeconomic policy). Next, there comes a description of a criterion, or economic goalspursued by the system under consideration. Finally there is assumed (also in amore or less explicit form) that the system behaves rationally, or optimally. Thismeans that the true trajectory provides an optimum to the criterion over all feasiblepaths starting at the same initial point.

The criterion is a scalar valued functional defined on all feasible paths (we donot consider here the multicriterial case). Usually it takes the form of a sum or anintegral of a certain expression (referred to as utility) over the trajectory. But sucha form immediately brings under consideration an important parameter which hasbeen in shadow up to now. This parameter is the time horizon (otherwise referredto as the planning horizon).

If this parameter is assumed to take a certain finite value, than no additionalmathematical difficulties arise, since the finite sums/integrals are quite well defined,yielding thus the correct criterion of optimization. But economic difficulties appear.Their nature can be explained using the consumption/investment dichotomy.

The latter is the model in which the profit obtained on each step has to be dividedinto the immediate consumption part and the one to be invested for increase in theproductivity capacities of the system which results in the increase of the futureprofits. The criterion is the integral consumption.

Given a certain finite planning horizon, the optimal trajectories of the model(under reasonable assumptions) behave as follows. When on earlier stages, usually

Preprint. The final version appeared in Nonlinear semigroups and infinite horizon

optimization. Idempotent analysis, 167–210, Adv. Soviet Math., 13, Amer. Math. Soc.,Providence, RI, 1992. MR 120379.

Typeset by AMS-TEX

1

2 S. YU. YAKOVENKO AND L. A. KONTORER

there is an optimal proportion between consumption and investment which is moreor less precisely maintained along the optimal paths. But this proportionality holdsno more when the model is close to the beginning or the end of the time interval.The starting segment is greatly influenced by the initial state, and since this stateis prescribed, there is no problem in such a deviation, at least from the point ofview of methodology. Completely different is the other extremity, towards the endof the planning horizon. The system behavior on final steps follows the ‘apres nousle deluge’ pattern: the investment almost vanishes.

Clearly, such a property of the model is quite reasonable within the frameworkof the finite horizon optimization, since the decrease in investments would result inshrinking future profits, but this future is beyond the planning horizon. Apparently,such features of the model have to be avoided somehow. There is a series of tricksfor this. The simplest one is to disregard the model as soon as its behavior begins tobe influenced by this terminal effect, but this is not so easy in the multidimensionalcase with more than two alternatives. Some other approaches were developed indifferent particular cases, but they rely heavily on certain properties of the model(convexity etc.), and require a thorough preliminary investigation of properties ofall the finite extremals; the most illustrating example is the turnpike theory.

Instead of finite horizon problems, one might speak about infinite horizon ex-tremals. But, on the contrast to the finite case, the infinite sums and improperintegrals sometimes (and even very often) happen to diverge. Using a kind of regu-larization procedures, one can try to transform the criteria to something convergent,but the justification of such manipulations hardly goes beyond the heuristic level.

An alternative strategy consists in introducing certain partial orders on all thepossible cost flow paths corresponding to all feasible paths (the cost flow path isthe sequence of scalars cT ∈ R, T ∈ N or T ∈ R, where cT is the value takenby the criterion on the initial T -step segment of the trajectory). Unfortunately,such partial orders (or, more correctly, binary relations) are not very naturallyintroduced,1 and the most popular among them are nontransitive (a short surveyof this theory is given below).

As an immediate consequence of this nontransitivity there arises the existenceproblem: whether there exist the ‘best’ elements with respect to those orders (here‘best’ means either majorizing or nonmajorizable). It often happens that some ad-ditional assumptions are necessary to prove the existence theorems. The convexity(or strict convexity) assumption is most important among them, and sometimesthe analysis follows the lines of the turnpike theory.

Therefore the economic problem as it was just stated is to develop a dynamicoptimization theory which would be independent on the choice of the planning hori-zon.

1.2. Mathematics. Stationary dynamic optimization problems in mathematicsusually come in one of the following frameworks.

1for example, try to compare the following three cost flow paths:

0, 2, 0, 2, 0, 2, . . .

2, 0, 2, 0, 2, 0, . . .

1, 1, 1, 1, 1, 1, . . .

NONLINEAR SEMIGROUPS AND INFINITE HORIZON OPTIMIZATION 3

The discrete time problem. Let X be a phase state of the model. It couldbe either a subset of a Euclidean n-space, or a discrete set, or even an abstracttopological space. The points of this set represent different states of the system.We assume that the model is deterministic, so the information about the states iscomplete and precise.

A trajectory or path is a sequence of states x = xt ∈ X : t = 0, 1, 2, . . . .Therefore the time variable t is assumed to be taking only nonnegative integervalues.

The feasible paths are usually defined recursively: there exists a multivaluedmap F : X → 2X and x is feasible if and only if xt+1 ∈ F (xt) for all t (the systemis stationary, so F is independent of t).

The criterion of optimization in the most general form is the sum

B(x) =∑t

b(xt, xt+1),

where

b(·, ·) : X ×X → R

is the utility function: the value b(x, y) is associated with the utility of transitionfrom the state x to the state y in one step (this function is also time independent).From now on we will call b(·, ·) the transition function.

These data allow to pose the stationary dynamic optimization problem in discretetime,

B(x)→ max, x0 = a ∈ X,

where a is a prescribed initial state, and the maximum is taken over all feasiblepaths starting at a.

In order to simplify the construction we will incorporate the feasibility restric-tions into the transition function by setting b(x, y) = −∞ if y /∈ F (x) (the infinitepenalty). Clearly, we shall pay for such a simplification by imposing certain condi-tions of semicontinuity on b, but all the constructions will become more transparent.Without further mentioning it we made the

Semicontinuity Assumption. The phase state is a separable topological space,and the transition function b(·, ·) is upper semicontinuous, that is, its hypograph(x, y, u) ∈ X ×X × R : u 6 b(x, y) is a closed subset.

In fact, the functional B is still undefined, since we did not point out the upperlimit for the summation. If N ∈ N is a finite integer, then one can introduce thefunctional

BN (x) =

N−1∑t=0

b(xt, xt+1).

Replacing B by BN , we obtain a correct finite horizon optimization problem of acommon type.

Proposition. Under the semicontinuity assumption if the phase state is compactand the horizon N finite, then the optimization problem

BN (x)→ max, x0 = a


always possesses a solution.

The difficulties arise when defining something like B∞.The above finite horizon problem admits a natural generalization. Let f : X → R

be any upper semicontinuous function. Then we may add to the criterion theterminal term f(xN ). The problem thus obtained will be of the form (we write theunabridged expression)

N−1∑t=0

b(xt, xt+1) + f(xN )→ max(1.1)

xt ∈ X, t = 0, 1, . . . , N, x0 = a ∈ X.(1.2)

A solution to the problem is an N + 1-tuple from XN+1. The set of all solutionswill be denoted by extrN (b, f). The above proposition holds also for this case.

This generalization points out another peculiarity of the infinite horizon case.Indeed, the latter does not admit any terminal terms besides almost meaninglessones like f( lim

t→∞xt). At the same time if we will think of the infinite horizon

optimization problem as a limit (in a sense) of regular finite horizon problems, theterminants may occur in the latter ones. Thus the ‘right’ definition has to keepsome traces of a terminant.

The continuous time case: variational problems. We have given thedetailed description of the framework of the discrete time case, since all the phe-nomena analyzed in the current paper can be studied within it. Nevertheless thecontinuous time case is also very important.

Now we assume that the phase space X is endowed with the structure of thelinear space or at least is a differentiable manifold. In this case we may define andconsider absolutely continuous trajectories of the form x : R ⊇ ∆ 3 t 7→ x(t) ∈ X.

The role of the transition function is played by a Lagrangean function L(x, v),which defines the integral functional∫ T

0

L(x(t), x(t)) dt, t <∞,

to which an optional term f(x(T )) may be added. A variational problem consists infinding the maxima of this criterion over all admissible paths starting at a certainpoint a ∈ X.

The continuous time version causes greater technical difficulties with the exis-tence problem for the finite horizon case. Usually one requires concavity of L in vtogether with a relatively rapid decrease to −∞ as ‖v‖ → ∞. We shall treat thiscase in its turn.

The continuous time case: optimal control problems. The variationalproblem as it was stated above, admits no constraints. Those are generally in-troduced within the framework of the control theory. Let the phase space X beagain endowed with the smooth structure and suppose that there is a controlleddifferential equation on it, having the form

x = F (x, u), x ∈ X, u ∈ U,

where U is a certain topological space of admissible controls.


This control problem defines admissible solutions, which satisfy the above equa-tion almost everywhere. The criterion for such problems usually takes the form

T∫0

g(x(t), u(t)) dt+ f(x(T )).

All the remarks made beforehand refer as well to this case. Actually, the formallyintroduced Lagrange function

L(x, v) = supu∈U g(x, u) : F (x, u) = v , sup ∅ = −∞,

permits to reduce the optimal control problem to the variational one, if one allowsfor functions taking the value −∞.

We do not dwell on the technical matters relevant to the continuous time casefor the two reasons. First, these difficulties are not the point of our analysis, so wemay always assume any additional regularity of the problem referring the readerto any source. The second reason is that we suggest below a generalization of thenotion of a continuous time stationary dynamic optimization problem so that itincludes both opportunities. Of course, some new questions immediately arise, butthey will be treated elsewhere.

1.3. The structure of the paper. The paper is organized as follows. Firstwe analyze some heuristic approaches which lead to a priori properties of infiniteextremals, if they exist. It turns out that different arguments lead to the samefunctional equation which in the discrete time case (the one we are dealing withalmost everywhere throughout the paper) looks as

(BE) supx∈Xb(x, y) + f(y) = λ+ f(x).

Here b is the transition function, f : X → R is an unknown function and λ is acertain unknown scalar.

Of course, there is nothing new in this equation which is simply a discrete versionof the stationary Bellman equation. Solvability of it was studied under differentcircumstances in different sources.

But the left hand side part of this equation may be considered as the result ofapplication of a nonlinear operator, which we suggest to call the Bellman operatorwith the kernel b, to the function f . Hence (BE) becomes something like theequation on eigenfunctions of an integral operator. And this is not merely ananalogy. In fact, there is a natural algebraic structure associated with (BE), whichis similar to the standard linear one except that there is no subtraction operationas the inverse to the addition (lacking the appropriate term and wishing to stressthe analogy, we call this structure ‘linear’ in the quotation marks, extending thisagreement on all the other terms relevant to the structure). The Bellman operatorspreserve the “linear” structure, and (BE) indeed becomes the spectral equationwith respect to it.

This structure was also known before (see the survey section of the paper).‘Spectral’ properties are extensively studied in other papers constituting this vol-ume. The key point was to link both theories together with the infinite horizon


optimization. As the result we have developed a kind of nonlinear (though ‘linear’)representation of a dynamic optimization problem in the functional space.

The ‘eigenfunctions’ of the Bellman operator generate infinite paths which arenatural candidates for being called infinite extremals. We analyze their propertiesand study all the solutions of (BE). It turns out that they are closely relatedto limit sets of a dynamical system in the space of the above mentioned ‘linear’representation.

Some points of the construction can be clarified if we omit the initial state afrom the formulation (1.1), (1.2), replacing it by the periodicity condition

x0 = xN .

An axiomatic theory is worth nothing if it gives no information about already ex-isting problems. One section of the paper relates the axiomatic definition of infiniteextremals via ‘eigenfunctions’ to the classical definitions of catching up optimal andovertaking trajectories. The main point is that if there are infinite extremals in theclassical sense, then they are necessarily generated by some ‘eigenfunctions’, butnot vice versa. Somewhat presumptuously we suggest arguments which indicatesomewhat artificial nature of the classical definitions, and which speak in favor ofmore invariant Bellman-based construction.

Another point is to highlight the nature of the convexity assumptions peculiarto the mathematical economics, from the point of view of the Bellman calculus.We show that the dynamical system in the representation space, associated withconcave transition functions, is dissipative: there is a unique point which is thelimit set for all the orbits of the representation. The spectral properties for theconcave case are also much simpler.

As for the continuous time case, we associate with any optimization problemwhich is regular enough, a continuous time dynamic system in the representationcase. In fact, this system is a generalization of the evolution semigroup for theHamilton–Jacobi equation corresponding to the variational problem. We provespectral properties of the semigroup and discuss some open questions.

The roots and sources of the theory presented here, are dispersed among manypapers and books, some of them about 30 years old. We made an attempt to trans-late some known facts and theorems into the language of ‘linear’ semigroups. Sincethe translation cannot precede the vocabulary, we violated the existing scientifictradition by placing the survey section at the end of the paper. Needless to saythat our list had to be incomplete both in the mathematical and the economicalparts.

This paper existed in the draft form since the time the short communication wassubmitted for publication (1988). Recently one of the authors (S.Y.) had had achance to meet prof. Arie Leizarowitz (The Technion, Haifa) and to get acquaintedwith the series of his very illuminating papers on this subject. Many of assertionsproved in the current article were proved by him using sometimes similar, sometimesdifferent technique. Nevertheless we decided to preserve the initial structure of thepaper in order to show how the entire area may be exposed from the different pointof view.

1.4. Acknowledgments. During the preparation of the paper and developingthe apparatus we enjoyed many fruitful conversations with different people. We


are grateful to S. N. Samborskiı and V. N. Kolokol’tsov for discussions concern-ing idempotent analysis, A. M. Rubinov, V. Z. Belen’kiı, V. M. Polterovich andV. D. Matveenko for numerous opportunities to talk about economical matters un-derlying the subject. Talks with Zvi Artstein and Arie Leizarowitz helped us to lookon the problems from a different point of view. V. P. Maslov and A. D. Tsvirkunencouraged us and made the research possible. We are greatly grateful to all ofthem.

§2. Motivations. Three problems with one solution.

2.1. Preliminaries. As it was already mentioned in the Introduction, there areseveral possible approaches to definition of the notion of infinite extremals. Usuallythis is done using certain binary relations on the space of cost flows which is thespace RN of infinite sequences of the form c = c0, c1, . . . , ct, . . . , ct ∈ R. Theimportant example is the binary relation in the space of cost flows, defined as

ct = c > c′ = c′t ⇐⇒∀ε > 0 ∃N = N(ε) ∈ N : ∀t > N ct > c′t − ε.

More strong binary relation is the Pareto partial order on RN, and some interme-diate concept is the asymptotical Pareto order

c > c′ ⇐⇒ ∃N <∞ : ∀t > N ct > c′t.

Each partial order in the space of cost flows generates the correspondent partialorder on infinite trajectories XN: there is defined the cost flow map

C : x 7→ c = C(x) : cN

=

N−1∑t=0

b(xt, xt+1),

and we may putx > x′ ⇐⇒ x0 = x′0, and C(x) > C(x′)

for any relation > on RN.The other possible binary relations on the space of cost flows are given below.

Here we only point out that in the paper by L. Stern [1] there is given a compendiumof almost all logically possible definitions together with some relationships amongthem.

After a certain binary relation > on the space of infinite trajectories is intro-duced, there appears the possibility to speak about ‘maximal’ elements. By thisone usually means those which majorize in the sense of the relation > all the othertrajectories starting at the same point. Another possibility is to consider nonma-jorizable trajectories, i.e. those which cannot be majorized by any path with thesame beginning. This alternative doubles the number of possible definitions.

Further development of these ideas leads to the important notion of overtakingcriterion suggested by D. Gale [30] and C. C. von Weizsacker [31]. This approachwill be discussed somewhat later in its relationship to the concept of optimalitysuggested in §3,4.

Such degree of arbitrariness in definitions leads to some doubts. Moreover, soonone discovers the fact that, due to nontransitivity of many of relations defined this


way (especially this refers to nonmajorizable elements) and the lack of antisymmetry(both the possibilities x > y and y > x may occur simultaneously) these ‘extremals’do not always exist.

Thus we are led to an attempt to look for another sort of definitions, based ondifferent ideas. We will describe three different approaches leading to the samefunctional equation (BE).

2.2. Infinite series maximization. Suppose that for some unknown reasons theseries

S(x) =

∞∑t=0

b(xt, xt+1)

is so good that it generates a reasonable criterion for optimization, that is, it eitherdiverges to −∞ for ‘bad’ trajectories x, or converges and, moreover, the optimalvalue

(2.1) f(x) = maxx : x0=x

S(x)

is a well-defined semicontinuous function. Since we are dealing with heuristics, wedon’t discuss conditions guaranteeing such a behavior.

In such situation it is clear that the function f(·) : X → X must satisfy thecondition

(2.2) ∀x ∈ X maxy∈X b(x, y) + f(y) = f(x).

Indeed, if the trajectory x with x0 = x, x1 = y yields the maximum in the definitionof f(x), then the trajectory y with yt = xt−1 does so with respect to f(y) andconversely.

Unfortunately, the case of convergence is quite unstable: if by chance the transi-tion function b(·, ·) generates such a converging criterion, then for any λ 6= 0, λ ∈ Rthe function b′ = b+ λ yields the series S′ which is:

— diverging to −∞ for any x if λ < 0, or— possibly converging for certain paths, but diverging to +∞ for those which

were optimal with respect to the initial criterion, if λ > 0.

But this instability also gives a chance for remedy: given an arbitrary transitionfunction b, one may expect that its average growth rate along ‘optimal paths’ isequal to a certain constant value λ ∈ R, which could be subtracted in order toobtain the above convergence. Substituting this modified transition function b− λinto (2.2), we get somewhat more universal equation

(2.3) ∀x ∈ X maxy∈X b(x, y) + f(y) = λ+ f(x).

Now we may forget all the assumptions made about convergence and look at(2.3) as the equation for the unknown function f and the unknown value λ havingin mind that if all the assumptions hold, then the result must satisfy (2.3).

2.3. Dynamic programming. Another attempt is based on the possibility ofadditing a terminant to the criterion of optimization. Suppose that instead of thefinite horizon problem

N−1∑t=0

b(xt, xt+1)→ max, x0 = fixe


we are solving another one

(2.4)

N−1∑t=0

b(xt, xt+1) + f(xN )→ max, x0 = fixe

with a certain function f . The idea is to find a terminant f such that the solution ofthe problem (2.4) would be independent on N in the following sense: if N ′ > N as

a new horizon and xt N′

t=0 is the corresponding solution, then the initial segmentxt Nt=0 is the solution to the problem with the horizon N .

The regular way to solve problems like (2.4) is provided by the dynamic program-ming principle. One has to construct the sequence of functions defined recursively:

(2.5) f0(x) = f(x), fs+1(x) = supy∈Xb(x, y) + fs(y), s = 0, 1, 2, . . . , N − 1

and afterwards construct the solution sequence

(2.6) xt+1 ∈ Arg maxy∈X

b(xt, y) + fN−t−1(y), t = 0, 1, . . . , N − 1.

The desired horizon independence of the solution means that the process (2.6)is in fact independent of N . It would be so if all the functions fs were independentof the index s or, at least, differ only by some constants. The latter condition holdsif there is certain λ ∈ R such that f0 = f satisfies (2.3) together with this λ. Thusto find an appropriate terminant means to solve (2.3).

2.4. Symmetries. The last approach is based upon the idea of transformationsof an optimization problem. Let b : X ×X be a transition function. Along with bwe may consider another transition function of the form

(2.7) b′(x, y) = b(x, y) + g(y)− g(x) + µ

where g : X → R is a continuous function, and µ is an arbitrary scalar.The optimization problem with the transition function b′ is equivalent to the

initial one in the following sense:

(1) The two problems

N−1∑t=0

b(xt, xt+1) + f(xN )→ max, x0 = fixe

and

N−1∑t=0

b′(xt, xt+1) + (f − g)(xN )→ max, x0 = fixe

possess the same solutions for all initial values;(2) If we consider either the fixed endpoints or periodic problems with the

boundary conditions correspondingly x0 = fixe, xN = fixe or x0 = xN ,then the replacement of b by b′ does not change the extremals at all.


It is so because b′ differs from b by the full difference (the discrete time analogof the full derivative), and the addition of the constant to the transition functionnever changes the extremals.

The main idea related to the notion of equivalence is the idea of invariance:the equivalent problem must have the same properties. In our case this is to beunderstood as the claim that the equivalent transition functions generate the sameinfinite extremals. Thus it may prove useful to look for another transition functionwhich would be equivalent to the initial one and at the same time would be simplerto analyze.

Such functions indeed do exist.

Definition 2.1. The transition function b(x, y) is called good nonpositive, if

∀x, y ∈ X b(x, y) 6 0, and

∀x ∈ X ∃y ∈ X : b(x, y) = 0.

Another way to say that b is good nonpositive is to write

∀x ∈ X maxy∈X

b(x, y) = 0.

The infinite extremals for a good nonpositive transition functions are quite easyto define.

Proposition 2.1. If b(x, y) is good nonpositive, then:1. For any infinite trajectory x the series

(2.8)

∞∑t=0

b(xt, xt+1)

monotonically converges either to a nonpositive value or to −∞.2. For any initial point x0 ∈ X there exists at least one infinite trajectory x

starting at x0 and such that the series (2.8) is identically zero.

Corollary 2.1. The optimization problem

∞∑t=0

b(xt, xt+1)→ max

always admits a solution provided that b is good nonpositive.A trajectory x solves this problem if and only if

b(xt, xt+1) = 0

(= max

y∈Xb(xt, y)

).

Thus the invariance principle leads us to the following task: find a good nonpos-itive function equivalent to a given one.

The latter problem means to solve the equation

∀x maxy∈Xb(x, y) + g(y)− g(x) + µ = 0


which is equivalent to (2.3) with λ = −µ.

Therefore we see that to find a transformation of the form (2.7) taking the initialtransition function b to a certain good nonpositive one b′ one has again to solve thesame functional equation.

There arises a natural question about other possible transformations of the op-timization problem, besides that of type (2.7). At least one such transformationalso exists. Namely, let H : X → X be one-to-one continuous mapping (the statevariable change). Then this change of variable results in constructing of the newtransition function

(2.9) b(x, y) = b(H(x), H(y)).

Clearly, such a transformation adds nothing new to understanding of the initialproblem: the trajectory x solves the new problem if and only if H(x) solves theinitial one and vice versa (if X is a discrete compact, that is, a graph, then such atransformation corresponds to re-enumeration of its vertices).

But these two types of transformation are in a sense the only possibilities: thereis a natural symmetry group associated with optimization problems of this kind,and this group is generated by transformations of the form (2.7) and (2.9). Moreprecise formulation is given below.

2.5. Summary. The above considerations together lead to the following conclu-sions.

1. Each transition function b(x, y) satisfying the semicontinuity assumption,gives rise to the Bellman equation (2.3).

2. Any semicontinuous solution (λ, f(·)) of the Bellman equation generates infi-nite extremals by means of the recurrent formula

(2.10) xt+1 ∈ Arg maxy∈X

b(xt, y) + f(y) .

Using this formula, one can construct an infinite extremal starting at any pointx0 ∈ X.

3. If no other data except the transition function is provided, then there is nomeans to distinguish between different solutions of the Bellman equation.

4. When studying the optimization problems of this kind, one has to pay atten-tion to the invariance of all procedures and definitions with respect to transforma-tions of the form (2.7), (2.8).

Remark. Note that the condition (2.10) includes only the ‘first differences’, unlikethe discrete time counterpart of the Euler–Lagrange equations

(2.11) xt ∈ Arg maxy∈X

b(xt−1, y) + b(y, xt+1) , t = 1, . . . , N − 1,

which is the ‘second order’ difference necessary condition for optimality. In otherwords, behavior of an infinite extremal depends only on its initial state (thoughthis state might not determine the path uniquely).


§3. Glossary of the ‘linear’ algebra.

3.1. Algebra. In this section we introduce the structure of idempotent semi-module on the space of extended real valued functions on X and list all necessarydefinitions. Since this subject is extensively covered by the other papers of thepresent volume, our exposition will be as short as possible.

The main semiring of extended reals is denoted by R = R∪−∞. This semiringis endowed with the two binary operations ⊕ and :

∀a, b ∈ R a⊕ b = max(a, b), a b = a+ b.

The metric on R is defined as dist(a, b) = | exp a − exp b| (we put exp(−∞) = 0).This metric generates the correspondent topology on R making the topological or-dered semiring of it. Note that both operations are commutative and associative,with ‘addition’ ⊕ being distributive with respect to ‘multiplication’ . Unfortu-nately, there is no ‘subtraction’ in R.

These structures are inherited by the space R(X) of R-valued functions on X,the latter bearing thus the structure of a semimodule over the semiring R:

∀f, g ∈ R(X), λ ∈ R (f ⊕ g)(x) = f(x)⊕ g(x), (λ f)(x) = λ f(x).

There is a subspace in R(X) consisting of continuous functions nowhere taking thevalue −∞. We denote it by C(X) in the usual manner.

The semimodule and semiring structures were independently introduced in manyworks on combinatorial mathematics and graph theory; see, for example, the paperby F. Gondran [8] and references there, [28] etc. Recently this and similar structureswere deeply investigated by V. P. Maslov, S. N. Samborskiı, V. N. Kolokol’tsove.a.[9],[10]. To make the analogy with the regular ring/module structure moretransparent, we shall use terms borrowed from linear algebra, putting them in thequotation marks.

Important note. One of the main features distinguishing the ‘linear’ space R(X)from a linear one is the possibility of ‘summation’ of more than a countable numberof terms (numbers, functions etc.): for any uniformly bounded family fα ∈ R(X)we may put (⊕

α

fα

)(x) = sup

αfα(x).

The above remark means that an idempotent analog to the space of integrablefunctions is simply the space of functions on X, bounded from above. We denoteit by M(X): if X is compact, then

R(X) = f : X → R ⊃M(X) = f : sup f <∞ ⊃C(X) = f : f is continuous on X and nowhere equal to −∞.

3.2. Operators. The main reason why the algebraic structures appear in theexposition is that the expression occurring in the left hand side part of the Bellmanequation may be considered as the value taken by a certain operator on the givenfunction.


Definition 3.1. Let b : X × X → R be a semicontinuous function (that is, thefunction having the closed hypograph). Assume that the set X is compact.

The Bellman operator with the kernel b is the operator B : C(X)→ R(X), definedby the formula

(3.1) ∀f ∈ C(X) (Bf)(x) = maxy∈X b(x, y) + f(y)

Important note. Actually, this definition is correct, provided that f is only semi-continuous. Moreover, the applicability of this definition may be significantly ex-tended, if we replace ‘max’ by ‘sup’ in the above formulation, allowing thus for allb ∈ R(X ×X), f ∈ R(X) bounded from above. We shall denote this extension bythe same symbol B.

Proposition 3.1. The Bellman operator is ‘linear’:

(3.2) ∀f, g ∈ R(X), λ ∈ R B(f ⊕ g) = Bf ⊕Bg, B(λ f) = λBf.

The property (3.2) in fact is characteristic for Bellman operators. If a certaincontinuity property holds for an operator B satisfying (3.2), than it can be rep-resented in the form (3.1) with an appropriate function b ∈ R(X × X) [9], [10].Another condition sufficient for possibility of such a representation is a ‘continuallinearity’ which means that (3.2) holds not only for two terms, but for any familyof terms:

(3.3) B

(⊕α

fα

)=⊕

Bfα

provided that this family is ‘summable’ (=uniformly bounded). In this case thekernel function b(x, y) can be reconstructing from the values taken by B on theδ-functions

δa(x) =

0, if x = a,

−∞ otherwise.

Indeed, b(x, y) = (Bδy)(x).

Remark. There can be formulated explicit conditions to be imposed on the kernelb for the associated Bellman operator to map the space C(X) into itself. This is ofno special interest to us, see [10].

3.3. The algebra of operators. The general principle is to consider an algebraicobject (the Bellman operator in our case) together with all objects possessing thesame algebraic properties. So we introduce the set EndR(X) of all ‘linear’ operatorson R(X) and the subset EndC(X) of those preserving the continuity of functions.

Operators from EndR(X) can be ‘added’ and ‘multiplied’ by scalars from R.Clearly, these two operations preserve the property (3.2) and (3.3). For any two op-erators B1, B2 ∈ EndR(X) their superposition B1 B2 again belongs to EndR(X),the latter set being thus a semialgebra over the semiring R.

The identical operator id belongs to EndR(X): its kernel is the function

δ(x, y) =

0, if x = y,

−∞ otherwise.


An operator C ∈ EndR(X) is called invertible, if there exists C−1 ∈ EndR(X)such that C C−1 = C−1 C = id. All the invertible operators form the groupInvR(X).

As in the case of linear operators, the two operators B, B are called conjugate,if there exist an invertible operator C ∈ InvR(X) such that

(3.4.) B = C−1 B CThis is an equivalence relationship on the semialgebra of Bellman operators.

Examples.1. For any continuous function g : X → R (g(x) 6= −∞) the operator

Dg : f 7→ f + g ⇐⇒ (Dgf)(x) = g(x) f(x)

is the invertible ‘diagonal’ operator with the inverse D−g.2. For any one-to-one map H : X → X the operator

PH : f 7→ f His invertible with the inverse PH−1 . This and the previous examples belong toEndC(X) if g is continuous and H is a homeomorphism.

3. If C = Dg, and B is a Bellman operator with the kernel b, then the conjugate

operator B = C−1 B C has the kernel

b(x, y) = −g(x) + b(x, y) + g(y),

which we have already met beforehand when discussing the notion of equivalentoptimization problems in §2. The same refers to the second of the above examples.

Definition 3.2. Two discrete time optimization problems with the transition func-

tions b(x, y) and b(x, y) are called equivalent if the associated Bellman operators B

and B are conjugate up to ‘multiplication’ by a scalar: ∃C ∈ InvR(X), λ ∈ R : B =λ C−1 B C.

Now we can explain why only the conjugation by matrices of the form Dg andPH can be considered.

Proposition 3.2 (V. N. Kolokol’tsov, 1987). An operator C ∈ EndR(X) is invert-ible if and only if it can be represented in the form Dg PH with some everywhere

finite function g ∈ R(X) and a certain one-to-one mapping H.

Actually, a Bellman operator C is invertible if and only if its kernel c(·, ·) satisfiesthe condition

∀x ∃!y : c(x, y) 6=∞.Therefore there appears a one-to-one mapping H : x 7→ y which will be denoted byTC . The optimization problem associated with an invertible operator is nothingmore then a deterministic discrete time dynamical system on the phase space X,generated by the transformations Ht, t ∈ Z, H = TC .

Remark. In the above exposition the space R(X) can always be replaced by M(X)provided that the kernel function is bounded from above.

Important note. The important point is that though the group InvC(X) is rel-atively small, it nevertheless acts transitively on the space C(X). Indeed, anyfunction f1 ∈ C(X) can be taken into any other f2 by the invertible operator Dg

with g = f1 − f2. But this is not true if the functions take the value −∞.


3.4. Topology. Up to now only algebraic properties were under consideration.Now we invoke topological matters. Our attention will be concentrated mostlyon the spaces M(X) and C(X) (recall that the former one plays the role of theLebesque L1-space from the standard functional analysis).

Definition 3.3. The Bellman operator (3.1) with the kernel b ∈ M(X × X) iscalled compact, if:

(1) the phase space X is a metric compact, and(2) the function b is continuous and nowhere taking the value −∞.

To motivate this definition, we claim the following. Any compact Bellman op-erator takes its values in the space of all continuous functions. Moreover, whenconsidered as the nonlinear operator acting on the Banach space C(X) endowedwith the usual norm ‖f‖ = max

x∈X|f(x)|, it takes any bounded subset of the latter

space into a set possessing the compact closure (pre-compact). The proof is basedon the two lemmas.

Lemma 3.1 (on uniform continuity of images). The image of the set M∗(X) =M(X) \ f ≡ −∞ by a compact Bellman operator B is uniformly continuous:

∀ε > 0 ∃δ > 0: ∀f ∈M∗(X), ∀x1, x2 ∈ Xρ(x1, x2) < δ =⇒ |Bf(x1)−Bf(x2)| < ε,

where ρ is the metric on X.

Proof. Being continuous on the compact set X ×X, the kernel is equicontinuous:

∀ε > 0 ∃δ > 0: ∀x1, x2, y ∈ X ρ(x1, x2) < δ =⇒ |b(x1, y)− b(x2, y)| < ε.

Therefore for ρ(x1, x2) < δ one has

Bf(x1) = supy∈X b(x1, y) + f(y) 6 sup

y∈X b(x2, y) + f(y) + ε = Bf(x2) + ε,

and the inverse inequality also holds: Bf(x2) 6 Bf(x1) + ε, whence the uniformcontinuity comes.

Corollary 3.1. A compact Bellman operator always has continuous values:

B (M∗(X)) ⊂ C(X).

Lemma 3.2 (on monotonicity). Any Bellman operator preserves the Pareto partialorder on M(X): if f1 > f2 on X, then Bf1 > Bf2.

Proof. The inequality f1 > f2 is equivalent to the identity f1 ⊕ f2 = f1. TheBellman operator is ‘linear’, hence Bf1 ⊕Bf2 = Bf1.

Corollary 3.2. Any Bellman operator is nonexpanding in the Banach space C(X).

Proof. If ‖f1−f2‖ 6 r, then f1 6 f2+r, f2 6 f1+r. Applying Lemma 2 and using‘linearity’ of B, we get Bf1 6 Bf2 + r and vice versa, whence comes the inequality‖Bf1 −Bf2‖ 6 r. Remark. Lemma 2 and Corollary 2 remain valid also for noncompact operators: inthis case the Corollary asserts that

supX|f1 − f2| 6 r =⇒ sup

X|Bf1 −Bf2| 6 r.


Theorem 3.1. A compact Bellman operator is continuous on the Banach spaceC(X) endowed with the norm ‖ · ‖ and takes any bounded subset of it into a pre-compact one, being thus the compact operator in the usual sense of this term.

Proof. The continuity of B is proved in Corollary 2. For any bounded subset U ⊂C(X) the set B(U) is uniformly continuous by Lemma 1 and uniformly boundedby the constant

maxx,y∈X

|b(x, y)|+ supU‖f‖.

The Ascoli–Arzela criterion of pre-compactness in C(X) is applicable, hence B(U)is pre-compact.

3.5. Summary. We have introduced the most important for us notion of theBellman operator and analyzed the role of continuity of the kernel in terms ofnonlinear functional analysis.

§4. ‘Eigenfunctions’.

4.1. ‘Spectral’ equation. As the main result, the heuristic arguments listed in§2 lead to investigation of the Bellman equation (3.1) which, using the notationintroduced in §3, can be written in the form quite similar to the spectral equationfrom linear algebra:

(4.1) Bf = λ f, λ ∈ R, f ∈M(X).

4.2. Quotient ‘projective’ space. The Banach space C(X) contains the 1-dimensional subspace of constants denoted by R. The quotient space C(X) =C(X)/R inherits the structure of a Banach space endowed with the quotient norm‖f(·)‖ = min

λ∈R‖λ + f‖. The equivalence class of an element f will be denoted by

f + R.The quotient space C(X) is obtained from C(X) by ‘projectivization’: elements

f and λ f are identified if λ is ‘nonzero’ (i.e. λ 6= −∞).The Ascoli–Arzela criterion of pre-compactness in C(X) implies the following

pre-compactness criterion for C(X):

Lemma 4.1. Suppose that X is a compact metric space.A family fα + R : fα ∈ C(X), α ∈ A is pre-compact in C(X), if the family

fα is equicontinuous.

Proof. If the family fα is equicontinuous, and X is compact, then the total os-cillation supX fα − infX fα is uniformly bounded over all α ∈ A. Therefore onemay add to each fα certain constant λα (say, λα = − infX fα) so that the new fam-ily λα + fα be uniformly bounded, therefore pre-compact by the Ascoli–Arzelatheorem. This immediately implies pre-compactness of fα + R in C(X).

Since any Bellman operator B commutes with the ‘multiplication’ by scalars,there can be defined the quotient operator B:

(4.2) B(f + R) = Bf + R,

which is a nonlinear operator. If B is compact, then by Lemma 1, §3 the quotientoperator B is the compact operator from the space C(X) into itself. Moreover,the image B(C(X)) is pre-compact in C(X).


4.3. Existence of ‘eigenfunctions’ for a compact Bellman operator. Nowwe can prove solvability of the ‘spectral’ equation (4.1).

Theorem 4.1. Any compact Bellman operator possesses at least one continuous‘eigenfunction’.

Proof. One has only to apply the Schauder fixed point principle to B: this is acompact operator taking a sufficiently big ball in C(X) into itself.

Thus there exists a fixed point f∗ + R ∈ C(X) for B:

B(f∗ + R) = f∗ + R,

which means that f∗ together with a certain λ ∈ R satisfies (4.1). The number λis finite, and f∗ is continuous by construction.

4.4. ‘Adjoint’ operators. To prove the uniqueness of the ‘eigenvalue’ for acompact Bellman operator, we need a ‘scalar product’ in M(X).

Definition 4.1. The standard ‘scalar product’ in M(X) is the ‘bilinear’ functional

〈f, g〉 = supx∈X

(f(x) + g(x)) =⊕x∈X

f(x) g(x).

In the same manner as this is done in the regular linear case, we define the‘adjoint’ operator B∗ to any Bellman operator B by means of the identity

∀f, g ∈ C(X) 〈Bf, g〉 = 〈f,B∗g〉 .

Proposition 4.1. The ‘adjoint’ operator always exists. Its kernel b∗ is given bythe formula

b∗(x, y) = b(y, x)

(the transpose matrix!).The ‘adjoint’ to a compact operator is also compact.

4.5. Uniqueness of the ‘spectrum’. We are starting to study the set of solutionsto the ‘spectral’ equation.

Theorem 4.2. The ‘eigenvalue’ for any compact Bellmen operator is unique.

Proof. Let λ be one of the ‘eigenvalues’ existing by virtue of Theorem 1. Since the‘adjoint’ operator B∗ is also compact, there is at least one ‘eigenvalue’ λ∗ for B∗.Denote the corresponding ‘eigenfunctions’ by f and f∗ respectively. Since both ofthem are finite, 〈f, f∗〉 > −∞. But by definition one has

λ 〈f, f∗〉 = 〈Bf, f∗〉 = 〈f,B∗f∗〉 = λ∗ 〈f, f∗〉 ,

therefore we get λ = λ∗. But this means that any element of the ‘spectrum’ of Bequals to any one of B∗, whence the uniqueness.

4.6. Corollaries and examples. For convenience we introduce the notation.The unique ‘eigenvalue’ for a compact operator B will be denoted by spec(B). Theset of the corresponding ‘eigenfunctions’ (the ‘eigenspace’) is denoted by es(B).


Corollary 4.1(I. V. Romanovskiı, 1967). Suppose that B is a compact operator.Then for any function f ∈ C(X) the iterates Btf, t = 0, 1, 2, . . . grow as an

arithmetic progression: there exists λ ∈ R such that

‖Btf − f − tλ‖ = O(1) for t→∞

(here the iterates Bt are defined as Bt = B Bt−1).

Proof. Take any f∗ ∈ es(B). Then Btf∗ = tλ, where λ = spec(B). Next, since allBt are nonexpanding, one has

‖Btf −Btf∗‖ 6 ‖f − f∗‖ = O(1),

which immediately yields the required estimate.

Corollary 4.2 (A. Leizarowitz [32], 1985). Any transition function continuous onthe compact phase state X can be represented in the form

b(x, y) = λ+ g(y)− g(x) + b′(x, y)

where b′(·, ·) is a good nonpositive function, and g is continuous on X.

The problem of finding a ‘useful representation’ (the term by A. Leizarowitz) wasreduced to the Bellman equation in §2. In fact, an operator with a good nonpositivekernel function is completely characterized by the following property: the identicalzero function 0 is the ‘eigenfunction’ for the associated Bellman operator. Moreover,the reduction to the good nonpositive form is just the transformation B 7→ B′ =D−1f B Df , where f ∈ es(B). Indeed, Df (0) = f , so B′(0) = λ+ 0.

Example 1. The spectrum may be not unique if any of the two conditions of com-pactness of Bellman operators is violated. Indeed, if B = Dg is a ‘diagonal’ opera-tor with the nonconstant function g, then (independently of compactness of X) the‘spectrum’ contains at least all the values of g: the delta-function δa(x) satisfies(4.1) with λ = g(a). This nonuniqueness of the spectrum is due to the discontinuityof the kernel b(x, y) = δy(x) g(x).

Example 2. Another condition of compactness being violated, the spectrum maybe not unique as well. Let X = Rn be the noncompact phase space and assumethat b(x, y) = Φ(y− x), where F : Rn → R is a concave function (as smooth as onewishes). In this case any linear function f(x) = 〈p, x〉 , p ∈ Rn∗ satisfies (4.1):

supy∈Rn

Φ(y − x) + 〈p, y〉 = supv∈Rn

Φ(v) + 〈p, x〉+ 〈p, v〉 = 〈p, x〉+ Φ∗(p)

where Φ∗(·) is the Legendre transform for −Φ. If Φ is not a linear function, thenthe set of points at which Φ∗ is finite, consists of more than one point, hence thespectrum is “multiple”.

Proposition 4.2. The ‘eigenspace’ consisting of ‘eigenfunctions’ with the same‘eigenvalue’, is indeed a ‘subspace’ (=subsemimodule of M(X)):

es(B)⊕ es(B) = es(B), R es(B) = es(B).


This means that together with f ∈ es(B) any function λ f = λ + f is alsoan ‘eigenfunction’. Therefore when analyzing nonuniqueness of ‘eigenfunctions’ wewill mean by this that es(B) cannot be generated by any single function.

Example 3. Even if an operator is compact, its spectrum being thus a singleton,the ‘eigenfunctions’ can be nevertheless multiple in the above sense. The simplestpossible case is the two-point set X = 1, 2 with the following transition matrixA = ‖aij‖, aij = b(i, j), i, j = 1, 2:(

0 −1−2 0

).

Indeed, in this case we have the compact operator (since all aij are finite), and boththe two functions

f1(1) = 0, f1(2) = −2 and f2(1) = −1, f2(2) = 0

belong to es(B) with spec(B) = 0, but f2 6= λ f1 for any λ ∈ R.The matrix case (that of discrete compact X) was extensively analyzed by

I. V. Romanovskiı [16], where he has given a complete description of ‘eigenfunction’in terms of maximal loops of the weighted graph representing the correspondingdiscrete optimization problem.

4.7. The vanishing viscosity technique. There is a transparent analogy be-tween the spectral theory of Bellman operators and that of positive linear operators.For example, the way the uniqueness of the spectrum was proved is just the sameas used in the demonstration of the Frobenius–Perron theorem on nonnegative ma-trices.

This analogy can be explained as follows. Let dx be any finite nonnegativemeasure on X such that open subsets have a positive measure (such the measureexists because X was assumed to be separable, hence there exists a countable densesubset; the measure can be constructing by placing a mass of 2−k at the k-th pointof this subset).

Then for any f ∈ C(X) one has

limh→0+

h log

∫X

exp

(f(x)

h

)dx = max

x∈Xf(x).

Consider the family of operators Bh approximating a compact Bellman operator Bwith the continuous kernel b(·, ·):

(Bhf) (x) = h log

∫X

exp(h−1(b(x, y) + f(y))

)dy.

Then Bh is the superposition Lh Ah L−1h , where Lhf = h log f , and Ah is thelinear (without quotation marks!) integral operator on C(X) with the continuouseverywhere positive kernel ah(x, y) = exp

(h−1b(x, y)

). Applying the Frobenius–

Perron theorem in its infinite-dimensional version [12], we conclude that there existsan eigenfunction ϕh ∈ C(X) for Ah, therefore fh = Lhϕh is an ‘eigenfunction’ forBh.


One can easily check that there is possible to pass to the limit for h→ 0+ alongsome sequence hk → 0+ in such a way that fhk

would be uniformly converging inC(X). Since Bh → B, the limit function satisfies (4.1).

Actually, the dependent variable substitution u 7→ h−1 log u is used to transformthe nonlinear Burgers equation

∂u

∂t=

(∂u

∂x

)2

+ h∂2u

∂x2

into the linear parabolic heat conduction equation

∂u

∂t= h

∂2u

∂x2

which can be explicitly investigated. The second order term in the nonlinear equa-tion is a small viscosity added to the Hamilton–Jacobi equation. Since the latterhas evidently related to optimization, there is nothing to wonder about applicabilityof such an approach in some other ways, see [9], [14].

§5. Infinite Extremals

In this section we give the definition of infinite extremals for the discrete opti-mization problem with the transition function b : X ×X → R.

5.1. Definition of infinite extremals. Within the framework of our approach,we associate the notion of an infinite extremal with the Bellman operator B itselfrather than with its kernel (the transition function). The properties of infinite ex-tremals established in subsequent sections will motivate the choice of the definitionsuggested. The key point is the formula (2.6) generating finite horizon extremalsvia the dynamic programming procedure.

Main definition 5.1. Let B ∈ EndC(X) be the Bellman operator and f ∈ es(B)

an ‘eigenfunction’. An infinite trajectory x = xt ∞t=0 will be called f -extremal, ifand only if

(5.1) ∀t = 1, 2, . . . xt ∈ Arg max b(xt−1, y) + f(y)

or, equivalently,

(5.2) ∀t = 1, 2, . . . b(xt−1, xt) = f(xt−1)− f(xt) + spec(B).

The set of all f -extremals will be denoted as extr∞(f , B). We shall say that x

is an infinite extremal without referring to the choice of f , if it is a f -extremal for

some f ∈ es(B); we shall denote

Extr∞(B) =⋃

f∈es(B)

extr∞(f , B).

Clearly, Extr∞(λ C−1 B C) = TC(Extr∞(B)) for any C ∈ InvC(X), whichmeans that the notion of the infinite extremal is invariant under inner automor-phisms of the operator semialgebra.

Theorem 4.1 and the solvability of the recurrent equation (5.1) imply the follow-ing statement.


Theorem 5.1. Let B ∈ EndC(X) be a compact Bellman operator with the con-tinuous kernel, and x0 ∈ X an arbitrary initial state. Then there exists at least oneinfinite extremal x starting at the point x0.

Now we list some evident properties of infinite extremals defined this way.

5.2. Generalized Euler–Lagrange condition.

Definition 5.2. We shall say that a trajectory x (either finite or infinite) satisfiesthe generalized Euler–Lagrange condition, if for any two numbers t1 < t2 and anysequence of states yt : t1 6 t 6 t2 with xti = yti , i = 1, 2 one has the inequality

t2−1∑t=t1

b(xt, xt+1) >t2−1∑t=t1

b(yt, yt+1).

In other words this means that the trajectory cannot be improved (with respectto the given criterion) by changing any finite number of its states.

In particular, if the trajectory x satisfies the generalized Euler–Lagrange condi-tion, then for any t > 0 (and t < N in the finite horizon case)

xt ∈ Arg maxy∈X

b(xt−1, y) + b(y, xt+1)

which in the case of smooth transition function implies

∂b

∂y(xt−1, xt) +

∂b

∂x(xt, xt+1) = 0

that is, the usual discrete time analog of the Euler–Lagrange necessary conditionsof optimality.

Proposition 5.1. Any infinite extremal satisfies the generalized Euler–Lagrangecondition.

The proof is straightforward using the Bellman optimality principle.

5.3. First versus second order. As this follows from the previous section, theEuler–Lagrange condition reduces itself to something like second-order differenceequation. Therefore to ‘determine’ such a trajectory one needs either two initialstates, or the initial and the final state.

For any fixed f ∈ es(B) the f -extremals are ‘determined’ only by the initialcondition, as it follows from (5.1). This property can be formulated as follows.

Proposition 5.2. Let x be any f -extremal, and x′ be another f -extremal with the

same ‘eigenfunction’ f such that x′0 = xN for some N ∈ N.Then the trajectory

xt =

xt for t = 0, 1, . . . , N,

x′t−N for t = N + 1, N + 2, . . . ,

is also f -extremal.


In other words, infinite extremals associated with the same ‘eigenfunction’ canbe pasted together.

Example 5.1. This is not true for different ‘eigenfunctions’. The most simple ex-ample is the case of the translation invariant transition function b(x, y) = Φ(y− x)on X = Rn, though this is a noncompact case.

For the associated Bellman operator all the linear functions belong to esλ(B)provided that λ is a finite value of the Legendre transform of Φ (see Example 4.2).

The corresponding infinite extremals are straight lines (or, more exactly, thevector-valued arithmetic progressions). In general, the vectors generating theseprogressions, can be different, so when pasted together, segments of different ex-tremals from the class Extr∞(B) can form ‘broken lines’ which do not satisfy eventhe Generalized Euler–Lagrange conditions.

5.4. Good trajectories. There is a useful weakening of the notion of infiniteextremal which does not involve any reference to a specified ‘eigenfunction’. Thisnotion was also suggested by Gale [30].

Definition 5.3. Let B be a compact Bellman operator with spec(B) = λ. Aninfinite trajectory x is called good, if∣∣∣∣∣

N−1∑t=0

b(xt, xt+1)−Nλ

∣∣∣∣∣ = O(1).

The following properties of good trajectories are evident.

Proposition 5.3.

1. Any f -extremal is a good trajectory. Therefore good trajectories always exist.2. Any trajectory differing from a good one by a finite number of states, is also

good.3. The notion of a good trajectory is invariant under ‘conjugation’: if x is the

good trajectory for B ∈ EndC(X), and B′ = λ C−1 B C, then x′ = TC(x) isgood for B′.

5.5. Examples. Here are some examples of trajectories from the class extr∞(B)which demonstrate some important features.

Example 5.1. Let X = 1, 2, 3 be a three point phase state with the transitionmatrix −100 −1 −100

1 −100 −100α β −100

, α, β < 1.

Then there is more or less evident (since −100 is almost −∞), that there areonly two infinite trajectories starting at the point 3 and satisfying the generalizedEuler–Lagrange conditions. The associated cost flows are

α, α− 1, α, α− 1, . . . and β, β + 1, β, β + 1, . . . .

Both trajectories are good, and for certain α, β neither of them does not overtakethe other one.


This problem possesses the unique ‘eigenfunction’

f(1) = 0, f(2) = 1, f(3) = max(α, β + 1).

Therefore for the case α > β+1 the first of the two trajectories is f -extremal, whilethe opposite inequality implies optimality of the second one. This answer coincideswith that given by the heuristic argument that the average cost flow must be greaterfor the optimal path: the average equals to α− 1/2 and b+ 1/2 respectively. Thusthe heuristics is justified.

Example 5.2. Now we consider the 5-point scheme with the transition matrix

∗ ∗ ∗ ∗ ∗10 0 ∗ ∗ ∗∗ 1 ∗ 2 ∗∗ ∗ ∗ 0 5∗ ∗ ∗ ∗ ∗

where stars stand for sufficiently large negative values (almost −∞).

In this case there are also two infinite trajectories starting at the point 3 andsatisfying the generalized Euler–Lagrange condition: 3, 2, 2, 2, . . . and 3, 4, 4, 4, . . . ;their cost flows are 1, 1, 1, . . . and 2, 2, 2, . . . respectively, and from a superficialpoint of view it is evident that the second one is more preferable.

But if you look for finite horizon optimal paths starting at the center point,you will find out that there are only two of them satisfying generalized Euler–Lagrange condition, namely, 3, 2, 2, . . . , 2, 1 and 3, 4, 4, . . . , 4, 5, with the formeryielding the greater value of the integral cost (11 versus 7). Therefore the finitehorizon approximation leads to the quite opposite answer.

The reason for this to happen is that the associated Bellman operator possessestwo independent ‘eigenfunctions’:

f−(1) = −10, f−(2) = 0, f−(3) = 1, f−(4) = ∗, f−(5) = ∗,and

f+(1) = ∗, f+(2) = ∗, f+(3) = 2, f+(4) = 0, f+(5) = −5.

The role of ‘eigenfunctions’ may consist in compensating possible disadvantages ofthe last step, so only the ‘recurrent’ states are to be taken into account (for moredetails see the next section). In our case the two trajectories correspond to thetwo ‘eigenfunctions’, and there is no way to decide between them without invokingsome additional arguments.

5.6. Conclusions. The notion of the infinite extremal generated by a certain‘eigenfunction’ of the Bellman operator leads to the object which possess all theintuitive apriori properties, including that of invariance. The unique problem isto choose between different solutions of the Bellman equation. Without any ad-ditional information about the optimization problem one cannot decide about the‘proper’ function, since the symmetry group of invertible Bellman transformationsacts transitively on the space C(X).


§6. ‘Projectors’ onto the ‘eigenspace’.

6.1. Extended data for an optimization problem: the terminant togetherwith the transition function. In this section we discuss ways of passing to limitin the finite horizon terminal optimization problem in order to obtain an appropriatedefinition of infinite extremals.

From §4 we may easily conclude that without loss of generality any compactBellman operator may be regarded as normalized to satisfy the condition

(6.1) spec(B) = 0.

For this one is to replace the initial operator B by the new one λ B, whereλ = − spec(B). As it follows from the above exposition, such a normalization doesnot affect any properties of the infinite extremals.

For an operator B satisfying (6.1) the orbit Bn f ∞n=0 of any function f ∈ C(X)is the bounded equicontinuous (uniformly continuous and bounded) family in C(X),therefore its closure is compact. (The same holds for any function R(X), f 6≡ −∞starting from n = 1.)

A transition to limit for N →∞ in the problem

(6.2)

N−1∑t=0

b(xt, xt+1) + f(xN )→ max, x0 = a

within the general spirit of the suggested approach means that a function f ∈ C(X)is to be found, which is an ‘eigenfunction’, i.e. f ∈ es(B), and which would inheritsome properties of the terminant f in (6.2). Since the operator B is assumed to be

fixed, the correspondence f 7→ f may be written in the operator form

(6.3) f 7→ f = ΩBf ;

sometimes the explicit dependence on B will be dropped out.The correspondence f 7→ Ωf is to satisfy the following natural list of conditions:

(1) ΩB is a ‘linear’ operator: ∀B ΩB ∈ EndC(X);(2) ΩB is invariantly associated with B: if B = λ C−1 B′ C, C ∈

InvC(X), λ ∈ R, then

(6.4) ΩB = C−1 ΩB′ C;

(3) ΩB is identical on the ‘eigenspace’ of B;(4) for any f ∈ C(X) the image ΩBf is an ‘eigenfunction’ of B.

The last two conditions together mean that ΩB is an idempotent projector fromC(X) onto es(B) ⊂ C(X).

Apriori it is not clear, whether such an operator exists or not. In fact, we provebelow that for any compact operator B there always exists a projector ΩB satisfyingall the above properties. Moreover, there can be formulated an axiomatic conditionsuch that its adding to the above list will imply uniqueness of the projector. Butwe would like to start with an important example, proving at the same time theexistence assertion on the projectors. Recall that from now on we assume thatspecB = 0.


6.2. The McKenzie projector. Let x = xt ∞t=0 be an arbitrary infinite goodtrajectory, and a ∈ X an arbitrary initial state.

Define the value

γ(a, f,x) =

lim supN→∞N−1∑t=0

b(xt, xt+1) + f(xN ) , if x0 = a;

−∞ otherwise.

From the definition of the good trajectory it follows that the lim sup exists.Next, define the function

f : a→ f(a) = supx γ(a, f,x) : x is a good trajectory .

The operator Γ = ΓB : f → f satisfies the following conditions.

Proposition 6.1. ΓB is the ‘linear’ operator, and

BΓB = ΓB ,

so ΓB is the projector onto the ‘eigenspace’ of B.The correspondence B 7→ ΓB is invariant under conjugations: if B = C−1 B′

C,C ∈ InvC(X), then ΓB = C−1 ΓB′ C.

Proof. The ‘linearity’ of the operator ΓB follows immediately from the definitions.For a fixed a ∈ X, x ∈ X∞ the value γ is ‘linear’ in f , as this is implied by theidentity

lim supk→∞

max(αk, βk) = max

(lim supk→∞

αk, lim supk→∞

βk

),

valid for all pairs of real sequences. Since lim supk→∞ αk = limk→∞⊕∞

j=k αj , thisis a consequence of the identity

limk

∞⊕j=k

(αj ⊕ βj) =

limk

∞⊕j=k

αj

⊕lim

k

∞⊕j=k

βj

.

Taking supremum over all x’s does not affect this ‘linearity’.

The straightforward reasoning proves that for f = Γf one has Bf = f . Indeed,if x = xt and y = yt , yt = xt+1, then by the definition of γ

γ(x0, f,x) = b(x0, x1) + γ(x1, f,y).

Taking supremum in both sides of this equality, one gets

ΓBf(x0) = supxγ(x0, f,x) = sup

x1,y b(x0, x1) + γ(x1, f,y) =

supx1

(b(x0, x1) + ΓB(x1)) = BΓBf(x0).

The invariance comes directly from the definitions.


The projector ΓB constructed this way plays an essential role in the classicaldefinition of infinite extremals. We shall call ΓB the McKenzie projector.

Note 6.1. The two conditions

ΩB ΩB = ΩB (the idempotent property),

B ΩB = ΩB ,

taken together, mean that ΩB is the projector onto the space of fixed points of B:

∀f ∈ C(X) ΩBf ∈ es(B), ∀f ∈ es(B) ΩBf = f.

Now we are proceeding with the main result of this section. To motivate it weneed to point out the following

Proposition 6.2. For any finite horizon optimization problem we have

extrN (B, f)|[0,N−1] = extrN−1(B,Bf).

In other words, to solve an N -step problem with a terminant f is equivalent tosolving the (N − 1)-step one with the terminant Bf .

Proof. This is the Bellman optimality principle verbatim.

Therefore the projector ΩB associating an ‘eigenfunction’ with any terminant fhas to satisfy the additional condition

(6.5) ΩB B = ΩB

if we want it to preserve the Bellman principle. Note that the latter conditionmeans that ΩB takes a constant value on any orbit of the operator B.

Theorem 6.1. Let B be a Bellman operator with the continuous kernel on a com-pact space X, normalized by the condition specB = 0.

Then there exists a unique projector ΩB : C(X)→ es(B) satisfying the above listof conditions and constant on the orbits of B in the sense (6.5).

Note 6.2. The algebraic conditions imposed on ΩB in the nonnormalized case lookas follows:

Ω2B = ΩB , ΩB B = B ΩB = λ ΩB ,

where λ = specB.

Proof. Existence. Let ωN (f) be the closure of the family Bnf∞n=N . We havementioned above that ωN (f) is compact in C(X), and ω1(f) ⊇ ω2(f) ⊇ . . . . Sothere is the non-empty intersection ω(f) =

⋂N>1

ωN (f).

Let Ωf be the ‘continual integral’, or the ‘sum’ of the uncountable number offunctions Ωf =

⊕ϕ∈ω(f)

ϕ or, using standard terms,

Ωf(x) = maxϕ(x) : ϕ∈ω(f)

ϕ(x)


Here max is correctly used since ω(f) is compact in C(X). We have the followingformula for Ωf :

(6.6) Ωf = limN→∞

∞⊕n=N

Bnf

or

(6.7) Ω = limN→∞

∞⊕n=N

Bn

The limits in (6.6) is taken in norm topology on C(X), and that in (6.7) in thestrong operator one.

From (6.7) we deduce all desired properties: ‘Linearity’ is evident; if B = C−1 B′ C and Ω′ = ΩB′ , then

C−1 Ω′ C = limN→∞

C−1 (

∞⊕n=N

Bn) C = limN→∞

∞⊕n=N

(C−1 B C)n

whence comes the invariance.If Bf = f then Bnf = f and Ωf = f . Next,

∀f ∈ C(X) BΩf = B limN→∞

∞⊕n=N

Bnf = limN→∞

∞⊕n=N

Bn+1f = Ωf

whence Ωf ∈ es(B), that is, Ω(C(X)) = es(B).

At last, ΩBf = limN→∞∞⊕n=N

Bn+1f = Ωf , therefore (6.5) holds.

Uniqueness. Let Ω satisfy all the conditions imposed on projectors togetherwith (6.5). Then

∀f ∈ C(X) Ωf ∈ es(B),

so Ω Ωf = Ωf . Therefore

Ωf = ΩΩf = Ω ( limN→∞

∞⊕n=N

Bnf) = limN→∞

∞⊕n=N

ΩBnf = limN→∞

∞⊕n=N

Ωf = Ωf

(we again used the idempotency of ‘addition’⊕

). So the desired uniqueness isestablished.

Note 6.3. The construction of the projector Ω is very geometric. The set ω(f) is theω-limit set of the orbit Bnf∞n=0 as it is defined in the theory of dynamical systems,see [15]: this is the set of accumulation points of the orbit Bnf in the functionalspace C(X). By quite general reasons ω(f) in invariant by B: if g ∈ C(X) is theuniform limit for a subsequence Bnkf , then Bg is the one for Bnk+1f . Moreover,Bω(f) = ω(f): for any g = limk→∞Bnkf there exists at least one accumulationpoint g′ ∈ ω(f) for the subsequence Bnk−1f such that Bg′ = g.


The operator B, being ‘linear’ on any set M ⊂ C(X), leaves its ‘barycenter’⊕M

ϕ =⊕ϕ∈M

ϕ invariant:

B⊕M

ϕ =⊕M

Bϕ =⊕B(M)

ϕ =⊕M

ϕ.

Hence we obtain the fixed point of B in purely topological terms.

Example 6.2. Consider the finite discrete compact X consisting only of five pointsX = 1, 2, 3, 4, 5 . Consider the transition matrix from the Example 5.2.

Let f = fi = f(i), i = 1, . . . , 5 be the zero function: fi = 0 ∀i.Then Γf(2) = Γf(4) = 0, Γf(3) = 2 (another values are of no interest).

Another simple calculation shows that Ωf(2) = 10, Ωf(4) = 5 and Ωf(3) = 11.So we have Ωf 6= Γf mod R, and deduce from there, that ΓB 6= Γ (see theorem

6.1).This fact results in what we have already seen: the unique Γf -extremal starts

at 3 is 3, 2, 2, . . . , while the unique Ωf -extremal is 3, 4, 4, . . . .

The importance of the projection Γ for the classical theory will be clear fromthe §8 below.

§7. Cyclic optimization and the ‘trace’ formula.

In this short section we give an explicit representation for the ‘eigenvalue’ of acompact Bellman operator in terms of maximal cycles.

7.1. Cyclic optimization. A natural way to get rid of the terminant in thecriterion as well as of the initial state is to consider the cyclic optimization problemof the form

(7.1)

N−1∑t=0

b(xt, xt+1)→ max, x0 = xN.

Denote the set of solutions to this problem by extrN (B) ⊂ XN . Evidently, this

set is invariantly associated with B: if B = λ C−1 B′ C, then extrN (B) =

TC(extrN (B′)).

7.2. The ‘trace’ of a Bellman operator. If B ∈ EndC(X), then the value

(7.2) trB = supx∈X

b(x, x) =⊕x∈X

b(x, x)

is called its ‘trace’.The ‘trace’ satisfies the following properties.1. It is ‘linear’ and invariant with respect to inner automorphisms:

tr(B1 ⊕B2) = trB1 ⊕ trB2, tr(λB) = λ trB,(7.3)

tr(C−1 B C) = trB(7.4)

for all Bi ∈ EndC(X), C ∈ InvC(X);2. The value of the problem (7.1) is equal to tr(BN ) = tr(B B · · · B) (N

times).


7.3. The ‘trace’ formula. We begin with establishing a formula which relates thespectrum of a compact Bellman operator to the trace of the ‘sum of the geometricprogression’.

Theorem 7.1. Let B be a compact Bellman operator with the zero ‘eigenvalue’spec(B) = 0.

Then

(7.5) tr

( ∞⊕n=1

Bn

)= 0.

Proof. Due to the invariance and ‘linearity’ of the trace without loss of generalitywe may put that the kernel b of the operator is good nonpositive.

This implies immediately that along any cyclic orbit x0, x1, . . . , xN−1, x

N= x0

the sum

N−1∑t=0

b(xt, xt+1)

is nonpositive, therefore the left hand side part of (7.5) is also nonpositive.Suppose that it is negative (less than a certain−ε < 0). Consider the multivalued

map x 7→ F (x) = y ∈ X : b(x, y) = 0 ⊂ X. Since b is continuous and X compact,then for any ε > 0 there is δ > 0 such that dist(y, F (x)) < δ implies b(x, y) > −εuniformly in (x, y).

Now take any infinite extremal, say generated by the identical zero ‘eigenfunc-tion’. Since X is compact, this extremal must have an accumulation point x∗ ∈ Xwhich means that there will be a pair of states xτ , xτ+k at least δ/2-close to x∗,hence δ-close to each other. In other words, the trajectory makes almost δ-closedloop.

By the choice of δ one has 0 = b(xτ , xτ+1) > b(xτ+k, xτ+1) > −ε, therefore byδ-changing of only one point of the trajectory we get the exactly closed loop withthe ε-close value:

b(xτ+k, xτ ) +

τ+k−1∑t=τ+1

b(xt, xt+1) > −ε,

therefore trBk > −ε and we have the contradiction with the negativity assump-tion.

7.4. Corollaries.

Corollary 7.1 [16]. The ‘eigenvalue’ of a compact Bellman operator is given bythe formula

spec(B) = supn>1

1

ntrBn

which is the maximal average value of all closed loops.

Remark. In fact, this corollary is much weaker than the theorem itself: to prove thelatter one has only to establish the estimate | trBn| = O(1) for the case spec(B) = 0.


Note 7.1. If X is a discrete compact consisting of #X points, the formula (7.5)may be strengthened:

(7.5’) tr

(#X⊕n=1

Bn

)= 0.

Indeed, the accumulation argument in this case is trivial: no later than after N =#X steps some of the states must be repeated in any infinite trajectory.

In fact, the maximal cycles (loops) in the discrete case correspond to ‘eigenfunc-tions’: if x is an N -periodic loop with the value equal to zero (we again considerthe case spec(B) = 0), then the function

(7.6) f(a) = lim supN

supy : y0=a,yN∈x

N−1∑t=0

b(yt, yt+1)

is an ‘eigenfunction’ with the zero ‘eigenvalue’, and the functions correspondingto disjoint maximal cycles are ‘linear’ independent. This problem was extensivelytreated in [16].

§8. A bridge to classical theory.

8.1. Classical notions of optimality. We recall here the notions of optimalitybased on the overtaking criterion as developed by Gale and von Weizsacker.

Let B be a compact Bellman operator with the transition function b : X×X → R.With any trajectory x = xt ∞t=0 we associate the cost flow c = c(x) = ct ∞t=1 ∈RN by the formula

cN

=

N−1∑t=0

b(xt, xt+1).

Definition 8.1.1. A trajectory x overtakes another trajectory x′ if both of them have the same

initial point and the associated cost flows c and c′ satisfy the condition

(8.1) lim inft→∞

(ct − c′t) > 0.

We shall denote this fact by the inequality x > x′. Sometimes, when it will be

necessary, the notation will contain the indication on the Bellman operator: xB> x′.

Iflim inft→∞

(ct − c′t) > ε > 0,

then we will say that x supertakes x′.2. A trajectory x is called overtaking, if it overtakes any other trajectory with

the same starting point.3. A trajectory is called weakly optimal, if it is not supertaken by any other

trajectory.

The condition (8.1) means that for any positive ε the value of a finite N -segmentof the path x is greater than that of x′ up to ε for all N large enough:

(8.2) ∀ε > 0 ∃N = N(ε) : ct > c′t − ε.

If in this definition ε can take the zero value, the notion of strict overtaking appears,but we will not dwell on these matters, see [1].


8.2. The McKenzie projector and the overtaking criterion. The McKenzieprojector was defined in §6. We recall its definition in somewhat different terms.We consider again the case of compact operators with spec(B) = λ.

Let Bf be the functional defined on infinite trajectories by the formula

(8.3) Bf (x) = lim supN∈N

N−1∑t=0

b(xt, xt+1)−Nλ+ f(xN

)

.

The McKenzie projector is the correspondence

(8.4) Γ : f 7→ Γf ; Γf(a) = supx : x0=a

Bf (x).

It was shown in §6 that Γ is indeed a ‘linear’ operator which takes its values in the‘eigenspace’ es(B).

The most important role in the classical infinite horizon optimization plays the

‘eigenfunction’ f0 = Γ0 (the value taken by the McKenzie projector on the identicalzero function).

Proposition 8.1. The function f0 is continuous on X provided that B is a compactoperator.

Indeed, all the ‘eigenfunction’ of a compact operator are continuous. The main

property of f0 is given by the following proposition.

Lemma 8.1. If a trajectory y is not an f0-extremal, than it does not overtake acertain trajectory x∗ starting at the same point.

Proof. Without loss of generality we may assume that the condition

(8.5) b(yt, yt+1) + f0(yt+1) = f0(yt) + λ

is violated from the very beginning (i.e. for t = 0), the difference between the leftand the right hand side of (8.5) being a certain negative −ε < 0. For simplicity wesubtract λ from the transition function, passing thus to the normalized case.

This means that for any finite N > 1 one has

f0(y0)− ε > f0(yN

) +

N−1∑t=0

b(yt, yt+1),

therefore by the definition of f0

lim supM→∞

M∑t=0

b(yt, yt+1) =

N−1∑t=0

b(yt, yt+1) + lim supM→∞

M∑t=N

b(yt, yt+1) 6

N−1∑t=0

b(yt, yt+1) + f0(yN

) < f0(y0)− ε,

and on the other hand there exists a certain trajectory x∗ such that

lim supN→∞

N−1∑t=0

b(x∗t, x∗t+1) > sup

x :x0=y0

lim supN→∞

N−1∑t=0

b(xt, xt+1)− ε/2 = f0(y0)− ε/2.


Finally we get

lim supN→∞

N∑t=0

(b(x∗t , x

∗t+1)− b(yt, yt+1)

)>

lim supN→∞

N−1∑t=0

b(x∗t, x∗t+1)− lim sup

N→∞

N−1∑t=0

b(yt, yt+1) > ε− ε/2 > 0,

which proves the claim, since for y to overtake x∗ it is necessary that this upperlimit were nonpositive.

As a corollary to this assertion, we obtain the following result.

Theorem 8.1. If an overtaking path exists, then it must be the infinite f0-extremal.

It is very likely that all the weakly optimal paths, if any, also belong to Extr∞(B),that is, they are generated by certain ‘eigenfunctions’.

8.3. Remarks and discussion. From the above exposition one can deduce thefollowing conclusions. First, the notion of overtaking is not invariant by conjuga-tions. This means that the relationship x > y may be affected when passing toanother transition function equivalent to the initial one. The McKenzie projectorrepresentation makes this point clear: the generating ‘eigenfunction’ is the Γ -imageof the identical zero function, while the latter one is not distinguished among theother functions.

In other words, to define a partial order of the above kind which would beinvariant, one needs to fix a certain terminant function and proceed like in the caseof the McKenzie projector.

Another remark concerns with the McKenzie projector itself. As this was pointedout in §6, there exists a unique projector onto ‘eigenspace’, which satisfies thecondition

Ω B = λ Ω.

In general the McKenzie projector does not satisfy it (e.g. for the case of Exam-ples 5.2, 6.2). This can be interpreted as a kind of noncommutativity of the twolimit transitions. When dealing with the infinite horizon optimization, one has topass to limit along any individual trajectory and to take supremum over all tra-jectories. Suppose that BN (x) is a certain finite-horizon criterion (for example,N−1∑t=0

b(xt, xt+1)). Then in general

supx

limN→∞

BN (x) 6= limN→∞

supxBN (x)

whatever this would mean.Both the two above arguments speak in favor of the definition of infinite ex-

tremals as the trajectories of the class

extr∞(B,Ωf)

provided that a certain apriori terminant f is given. But all the objections disappearif we consider a special case of operators having one-dimensional ‘eigenspace’, that


is, possessing a unique ‘eigenfunction’ up to addition of a constant. In this caseone can take this ‘eigenfunction’ as the generator for extremals with this choicebeing invariant by conjugations. Moreover, in such a situation all projectors differonly by scalar ‘factors’ which does not affect the construction of infinite extremals.Therefore it is this case for which the overtaking and the Bellman definitions ofinfinite extremals agree with each other. Later we will show that Bellman operatorswith strictly concave kernels (the case peculiar to economics) actually possess theunique ‘eigenfunctions’. This is the reason why it was the concave case which wasmost extensively investigated.

§9. Concave kernels and dissipative semigroups.

Now we turn to the most investigated case of discrete time optimization problemswith concave and strictly concave transition functions. For the latter case thereexists a complete description both for infinite extremals and the dynamics generatedby the associated Bellman semigroup in the functional space. This dynamics turnsout to be dissipative: this means that after normalization any orbit Btf convergesto a unique ‘eigenfunction’ of the operator.

9.1. The framework. From now on we will assume that:

(1) The phase space X is a convex compact subset of a finite-dimensional Eu-clidean space Rn;

(2) The transition function b(·, ·) is continuous and strictly concave on X ×X(the strict concavity of a function f on a set X means that for any x, y ∈ Xand for any α, 0 6 α 6 1 f(αx + (1 − αy)) > αf(x) + (1 − α)f(y) withthe strict inequality taking place for for all α 6= 0, 1);

(3) The restriction on the diagonal b(x, x) attains its maximum in a certaininterior point x∗ of X.

From the strict concavity it immediately follows that the point defined by thethird condition is unique.

9.2. The equilibrium prices and strictly nonpositive kernels.

Lemma 9.1. Under the above set of assumptions there exists a linear functionalp ∈ Rn∗ (called the equilibrium price system for certain economic reasons) such that

(9.1) ∀x, y ∈ X b(x, y) 6 b(x∗, x∗) + 〈p, y − x〉

(here 〈·, ·〉 stands for the usual scalar product in Rn).

Proof. This fact is a consequence of the standard separation theorem in Rn: the(convex) interior of the hypograph of the function b in Rn × Rn × R does notintersect the subspace x = y, z = b(x∗, x∗) hence there exists a linear hyperplanewhich also does not intersect the interior of the hypograph and contains the abovesubspace. Due to the interiority condition for x∗, this hyperplane is not vertical,therefore it has the equation of the form z = b(x∗, x∗) + 〈p, y − x〉 for certain linearfunctional p.

Corollary 9.1. In the assumptions of Lemma 9.1 the associated Bellman operatoris equivalent to an operator with strictly nonpositive kernel

(9.2) ∀x, y ∈ X b(x, y) 6 0, b(x, y) = 0 ⇐⇒ x = y = x∗.


Proof. Indeed, we make the transformation

b 7→ b(x, y)− 〈p, y〉+ 〈p, x〉 − b(x∗, x∗).

The last condition holds since otherwise we would have a segment along which thetransition is exactly linear; this contradicts the strict concavity.

Corollary 9.2. The kernel (9.2) in fact satisfies the more strict condition:

(9.3) ∀ε > 0 ∃δ > 0: dist(x, x∗) > δ =⇒ ∀y ∈ X b(x, y) 6 −ε.

Proof. This comes from the continuity of the kernel (9.2).

Corollary 9.3. If the optimization problem possesses the transition function sat-isfying the main assumptions, then any good trajectory x converges to the pointx∗:

(9.4) limt→∞

xt = x∗.

Proof. Indeed, otherwise by the corollary 9.2 the sum∑∞t=0 b(xt, xt+1) taken over

x diverges to −∞.

9.3. Overtaking trajectories in the strictly concave case. Since in thestrict concave case all the good trajectories converge to the same limit (and theother trajectories are not worth consideration), we have the optimization problemwith almost fixed endpoints: the condition (9.4) means that the right endpoint ispermanently fixed ‘at infinity’. Since such problems are invariant by conjugations(see §2), one can justify the transition to a conjugate operator (9.2).

Theorem 9.1. Let B be a Bellman operator satisfying the main assumptions.Suppose that there are two good infinite trajectories x,y such that one of themovertakes the other:

xB> y

(in particular this means that they have the same initial point).

If C is any invertible operator, then the overtaking property is invariant by con-jugation by C:

TC(x)C−1BC> TC(y)

Proof. Without loss of generality we may think of C as being purely diagonaloperator, therefore for the associated cost flows corresponding to B and C−1 B Cdiffer by the expressions f(x

N) − f(x0) and f(y

N) − f(y0) respectively. Since the

function f is continuous, and both xN

and yN

have the common limit x∗, whilex0 = y0, this modification does not change the limit condition (8.1) from thedefinition 8.1.


Corollary 9.4. If B is a Bellman operator satisfying the strict concavity assump-tions, then there always exist overtaking and weakly optimal trajectories for thecorresponding optimization prblem, starting at any point of the phase space.

Proof. Theorem 9.1 implies that when speaking about good trajectories, withoutloss of generality one can replace the initial problem by any equivalent one. There-fore we can put the transition function be good nonpositive.

Take any infinite trajectory x with the identical zero cost flow. Since the costflow of any other trajectory is nonpositive, it is evident that:

(1) x overtakes any other trajectory;(2) x is not supertaken by any other trajectory.

Therefore, returning back to the initial transition function, we conclude thatx overtakes and is not supertaken by any good trajectory. On the other hand,trajectories which are not good, cannot supertake the good ones and are alwaysovertaken by them.

9.4. Dissipative dynamical semigroup. Up to now we have discussed theproperties of infinite extremals in discrete time optimization problems with strictlyconcave transition functions. But this concavity implies certain very characteris-tic properties of the dinamical system on C(X) generated by the correspondingBellman operator

(9.5) f 7→ Bf.

As usual, we will assume that the operator B is normalized so that spec(B) = 0.

Theorem 9.2. Suppose that a compact normalized Bellman operator B satisfiesthe strict concavity assumptions.

Then:

(1) There exists a unique continuous concave ‘eigenfunction’ f ;

(2) Any orbit Btf converges to f in the uniform topology:

∀f ∈ C(X) limt→∞

‖Btf − f‖C(X) = 0.

(3) The iterates Bt converge to a ‘rank 1’ operator B; the kernel of this limit

operator can be represented as b(x, y) = f(x) g(y).

Proof. 1. Consider the two sequences of functions

fN

(x) = maxx : x0=x, xN

=x∗

N−1∑t=0

b(xt, xt+1),

gN

(y) = maxx : x0=x∗, xN

=y

N−1∑t=0

b(xt, xt+1).

Clearly, one has the following monotonicity:

fN+1

(x) > fN

(x), gN+1

(x) > gN

(x).


Indeed, any trajectory yielding maximum in the definition of fN

can be augmentedby the final step (x∗, x∗) which adds zero to the value of the criterion, leaving the(N + 1)-step path admissible with respect to the boundary conditions. The sameprocedure can be applied to the other sequence (augmentation by the trivial initialstep).

On the other hand, both sequences are bounded from above; hence there are

uniform limits f , g. All the functions (including the limit ones) are concave andeven strictly concave on X.

Finally, one has the identities

BfN

= fN+1

, B∗gN

= gN+1

following directly from the definition. Passing to limit, one obtains two ‘eigenfunc-tions’

Bf = f , B∗g = g.

2. Any trajectory solving the finite horizon problem

(9.6)

N−1∑t=0

b(xt, xt+1) + f(xN

)→ max, x0 = x ∈ X

has to come sufficiently close to the point x∗ provided that the time horizon N islarge enough:

∀δ > 0 ∃N ∈ N : x ∈ extrN

(B, f) =⇒ ∃t, 0 6 t 6 N : ‖xt − x∗‖ 6 δ.

Indeed, the value of the problem (9.5) is as close to

(9.7) f(x) + maxy∈X

(g(y) + f(y))

as we wish provided that N is large enough. On the other hand, if the entiretrajectory x is outside the δ-ball centered at x∗, then the value of (9.5) does notexceed

−Nδ + miny∈X

f(y)

and for N large there is a contradiction with (9.7) which gives the estimate in-dependent of N . Moreover, for any given δ the above reasoning proves that onlya bounded number of points of the trajectory x can be outside the ball with thebound depending on δ, f and uniform over all the initial points x ∈ X.

3. Therefore there must be a moment t such that both t and N − t are large andxt is close to x∗. Hence the value of the first segment is fairly approximated (for

N large) by f(x), the value of the second segment is close to g(xN

), and, since theright endpoint is free, the value of the entire problem (9.5) tends to

f(x) + maxy∈Xg(y) + f(y) .

4. The uniqueness of the ‘eigenfunctions’ f , g follows from the limit assertion.

Problem. In general, the uniqueness itself cannot guarantee the dissipative behaviorof the system. Is it true, that convergence of all the iterations Bt to a certain ‘rank1’ limit implies existence of overtaking extremals?


§10. Continuous time case.

The theory established above for discrete time optimization problems can bereformulated to cover the continuous time case. In particular, we are interested inproblems of the form

(10.1)

∫ T

0

L(x(t), x(t)) dt+ f(x(T ))→ max,

x(0) = a, ∞ > T ∈ R, x(·) is absolutely continuous on [0, T ].

In order to do such a reformulation we have to assume certain regularity of theintegrand L(x, v) independent of t. We assume that the global maximum in theproblem (10.1) is attained on an absolutely continuous arc for any initial conditiona and any finite time horizon T <∞ (there could be formulated explicit conditionson L guaranteeing such a behavior, but we are not interested in them).

Definition 10.1. For any T we define the Bellman operator BT : C(X) → C(X)as follows: (

BT f)

(a) = maxx(·)

∫ T

0

L(x(t), x(t)) dt+ f(x(T ))

provided that the maximum is attained.

Clearly, BT is indeed the Bellman operator with the kernel

bT (x, y) = maxx(·)

∫ T

0

L(x(t), x(t)) dt : x(0) = x, x(T ) = y

.

Since the Lagrange function L is independent of time, we have the followingfundamental property:

(10.2) BT BS = BT+S

which means that the family of operators BT constitutes a one-parameter semi-group, if one puts B0 = id by definition.

Definition 10.2. A function f ∈ C(X) is called an ‘eigenfunction’ for a Bellmansemigroup Bt, t > 0, if

(10.3) ∀t > 0 Btf = f + tλ

for a certain λ ∈ R which is called the ‘eigenvalue’ of the semigroup.

The difficulty of the continuous time case which distinguish it from the discretetime one is absence of the generating element. Nevertheless the main result claimingexistence of the ‘eigenfunction’ in the special case, still holds.

Definition 10.3. A family Bt of Bellman operators is called the continuous semi-group, if

(1) ∀t > 0 Bt ∈ EndC(X) is a compact Bellman operator;(2) Bt Bs = Bs Bt = Bt+s;(3) limt→0+ ‖Btf − f‖ = 0 for any f ∈ C(X).


Theorem 10.1. A continuous semigroup Bt possesses a continuous ‘eigenfunc-tion’.

Proof. Consider the sequence tk = 1/2k and the corresponding operators Bk = Btk .Each of them is compact by definition, therefore it possesses an ‘eigenfunction’fk ∈ C(X) : Bkfk = fk + λk, k = 0, 1, . . . . Since Bk Bk = Bk−1, we have:

(1) each fk is an ‘eigenfunction’ for all Bj , 0 6 j 6 k;(2) all fk belong to a uniformly continuous subset in C(X) (actually, the latter

is the image of B0);(3) 2λk = λk−1.

Hence there exists a converging subsequence of the form fki = fki + ci → f∞.

By construction, for infinitely many values of k one has Bkf∞ = tkλ + f∞, where

λ = λ0. Therefore for all the binary rational values of t Btf∞ = tλ + f∞. Bycontinuity of the semigroup the latter identity holds also for all nonnegative realt.

In the same manner as this was done in the discrete time case, Theorem 10.1 pro-vides a background for developing the theory of continuous time infinite extremals,parallel to that for the discrete time case.

The continuity assumptions can be verified in several cases, the most importantof them is the case of the Lagrange function being strictly concave in all its variablesand having a compact domain

domL = (x, v) ∈ Rn × Rn : L(x, v) > −∞.

For details see [29], [32], [17].

Remark. An interesting question (in fact, a whole series of them) arises when ana-lyzing connections between the abstract definition of semigroup and the particularcase of semigroups generated by integral functionals of the form (10.1). There areexamples of continuous semigroups which are not generated by any function L. Thesecond group of problems concerns differentiability of the Bellman semigroup Bt inthe time variable t: it is likely that in the same manner as the usual linear semi-groups of operators are differentiable almost everywhere, the Bellman semigroupsare (at least, under some reasonable assumptions). The last (but not least) problemconcerns the possibility to embed a Bellman operator in a continuous semigroup:for a givenB ∈ EndC(X) find a semigroup Bt so that B1 = B.

Some of this questions have been partially answered, but these topics go beyondthe scope of the present article.

§11. The survey and concluding remarks.

11.1. Overview of the reference list. The algebraic structures of semiring andsemimodule associated with combinatorial and graph optimization problems wereindependently discovered many times: some of the sources are listed in [8]. Therecent publications [9], [19],[10], [13] on this subject treat these structures in moredetails paying attention to functional-theoretic properties of the ‘linear’ operations⊕,.

Together with our standard assumptions = +, ⊕ = max there can be otherpairs of operations satisfying axioms of a commutative distributive semiring. The


most important of them is the pair ⊕ = max, = min (see [9], [19]). Theseoperations are used in the network flow theory.

Our main technical result, the existence of ‘eigenfunctions’, also has a long his-tory. In 1967 it was proved by I. Romanovskiı [16] for the case when the phasespace X is a discrete compact (the ‘matrix’ case) using combinatorial argumentsand the duality theory for the linear programming problem

λ→ inf

bij + fj 6 fi + λ, i, j = 1, . . . , n

−∞ 6 fj < +∞, λ ∈ R.

In the same paper there was introduced the notion of maximal loop and the struc-ture of the ‘eigenspace’ was analyzed.

Another paper by Romanovskiı [11] deals with the general case of compact phasespace X. He proved the asymptotic formula for iterates

Btf = tλ+O(1), t ∈ N, t→∞

using only an approximation technique. The paper also reveals connections of thespectral problem to the problem of mass transfer.

The most general case of the spectral problem was investigated in papers byP. N. Dudnikov, S. N. Samborskiı e.a. [33], [20]. They analyze the axiomaticsof idempotent semimodules and formulate conditions to be imposed on the pairof operations sufficient for discrete endomorphisms to have ‘eigenfunctions’. Thenondiscrete compact case was investigated using non-standard analysis.

The contents of §§5,6,8 is new (as far as we know). The idea of cyclical optimiza-tion goes back to Romanovskiı, and the ‘trace’ formula can also be found in [11].Actually, this paper, abundant with ideas, is rather difficult for understanding.

The idea to use Bellman principle to generate infinite extremals is rather com-mon: see [21] where the sum of an infinite series was used for this purpose. Anothersources: [4], [22].

An idea to transform the transition function by adding a full difference (fulldifferential to the integrand) in order to describe infinite extremals was exploitedin [5], [6].

Another way to approximate the Bellman operator by strict contractions

Bδf = maxy∈X

b(x, y) + δf(y), 0 < δ < 1

was suggested in [23]. This approach corresponds to investigation of the well-posedproblem

∞∑t=0

δtb(xt, xt+1)→ max, x0 = fixe .

The literature devoted to optimization problems with concave kernels is immense:without going to details we mention here the articles [2], [3], [7], [24]–[26]. In [17]there is discussed nonuniqueness of the ‘eigenfunction’ in concave but not strictlyconcave case arising in mathematical economics.

11.2. Miscellaneous topics. Concluding the paper, we mention some relevantresults. The detailed exposition will appear elsewhere.


11.3. The homogeneous case: von Neumann–Gale models. One of the verypopular models in mathematical economics is reduced to the following optimizationproblem. Let K ⊂ Rn+ ×Rn+ be a closed convex cone containing no elements of theform (0, x), x ∈ Rn+ \ 0 . Fix any initial state x0 ∈ Rn+ and any terminal criterionp ∈ Rn∗+ and consider the problem

〈p, xN〉 → max,

(xt, xt+1) ∈ K, t = 0, . . . , N − 1, x0 = fixe .

This finite horizon problem is well posed, but the question is how one can defineinfinite horizon extremals in this case. There is a definition based on the Paretopartial order on Rn+, and the existence theorem is proved under some additionalassumptions about the cone K.

There is possible to reduce the problem to the standard Bellman framework byletting the phase space X be the standard simplex σ = x ∈ Rn+ : x1 + · · ·+xn = 1and introducing the transition function

b(x, y) = maxλ∈R1

+

λ : (x, λy) ∈ K , x, y ∈ σ.

It turns out that under reasonable assumptions this system is dissipative inour sense, thus there exists the natural way to define infinite extremals (or, moreprecisely, their projections on the simplex), and this definition coincides with theclassical one [34].

The analysis is based on ideas suggested in [18].

11.4. Quasicompact case. Most of the theory exposed above for the compactBellman operators goes almost without modifications for operators which them-selves are not compact, but which have compact powers.

11.5. Game theoretic interpretation of the projector Ω. The projectorΩ: C(X)→ es(B) introduced in §6 and the corresponding infinite extremals admitsinterpretation in terms of the game theory. Without going into details we will sayonly that an inifinite trajectory x ∈ extr∞(Ωf,B) is the path which has to befollowed while solving the optimization problem

N−1∑t=0

b(xt, xt+1) + f(xN

)→ max, x0 = fixe

with the apriori unknown time horizon N .

11.6. Continuous time semigroups. The concluding Remark in §10 lists someproblems arising in the theory of continuous semigroups. We can add here thatcontinuous semigroups on a discrete compact phase space can be easily described:the associate matrices must have the form

‖btij‖ =

tλ1

tλ2. . .

tλn

with all the other elements being equal to −∞.


11.7. Hamilton–Jacobi theory. In the continuous time case variational problem(10.1) there is a relationship between the ‘eigenfunctions’ of the Bellman semigroupand Lagrangean invariant manifolds for the associated Hamiltonian system

x =∂H

∂p, p = − ∂H

∂x

withH(x, p) = max

v∈Rn 〈p, v〉+ L(x, v) .

The most simple case admitting a complete description is the case of concave func-tion L. If the hamiltonian H turns out to be smooth, then the function f ∈ C2(X)is the ‘eigenfunction’ for the Bellman semigroup if and only if its gradient graph

(p, x) ∈ Rn × Rn∗ : p =∂f

∂x

is the n-dimensional invariant manifold for the Hamiltonian system, which passesthrough a saddle-type singular point of it. For more details see [17].

References

1. Stern L. E., Criteria of optimality in the infinite-time optimal control problem, Journal of the

Optimization Theory and Applications 44 (1984), no. 3, 497–508.2. Brock W. A. and Haurie A., On existence of overtaking optimal trajectories over an infinite

time horizon, Mathematics of Operations Research 1 (1976), 337–346.

3. McKenzie L., Turnpike Theory, Econometrica 44 (1976), no. 5, 841–866.4. Belen’kiı V. Z., Objective functionals in stationary economic dynamic models, Modelling of

exchange mechanisms in economic systems, Central Economics and Mathematics Institute

research report, Moscow, 1979, pp. 93–121, (in Russian).5. Carlson D. A., A Caratheodori–Hamilton–Jacobi theory for infinite horizon optimal control

problems, Journal of the Optimization Theory and Applications 48 (1986), no. 2, 93–121.

6. Carlson D. A., On the existence of catching up optimal solutions for Lagrange problemsdefined on unbounded intervals, Journal of the Optimization Theory and Applications 49

(1986), 207–225.7. Haurie A., Existence and global asymptotic stability of optimal trajectories for a class of

infinite horizon non-convex systems, Journal of the Optimization Theory and Applications

31 (1980), no. 4, 515–533.8. Gondran F., Path algebra and algorithms, Combinatorial programming: methods and appli-

cations, ed. by B. Roy, N. Y., 1975, pp. 137–148.

9. Maslov V. P., Asymptotic methods for solving pseudodifferential equations, Nauka, Moscow,1987, (in Russian).

10. Kolokol’tsov V. N. and V. P. Maslov, General form of endomorphisms in the space of con-

tinuous functions with the range in a commutative semiring, Doklady Akademii Nauk SSSR295 (1987), no. 2, 283–287, (in Russian; transl. in Soviet Math. Dokl.).

11. Romanovskiı I. V., Asymptotic behavior of a discrete deterministic process with the continual

phase state, Optimal Planning 8 (1967), 171–193, (Novosibirsk, Nauka publ.).12. Krasnosel’skiı M. A. e.a., Positive Linear Systems, Nauka, Moscow, 1985, (in Russian).

13. Maslov V. P., On a new principle of superposition for optimization problems, Uspekhi Mat.

Nauk 42 (1987), no. 3, 39–48, (in Russian; transl. in Russian Math. Surveys).14. Crandall M. G. and P. L. Lions, Viscosity solution of Hamilton–Jacobi equations, Transactions

of th American Mathematical Society 277 (1983), 1–42.15. Nitecki Z., Differentiable Dynamics, The MIT press, Cambridge, MA and London, England,

1971.

16. Romanovskiı I. V., Optimization of stationary control of discrete deterministic process ofdynamic programming, Kibernetika (Kiev) 2 (1967), 66-78, (in Russian).


17. Yakovenko S. Yu., Fixed points of Bellman semigroup action and invariant manifolds for

Hamiltonian subdifferential equations, Avtomatika i Telemekhanika 6 (1989), 43–52, (in Rus-sian; transl. in Automation and Remote Control).

18. Tsvirkun A. D. and S. Yu. Yakovenko, Lyapunov functions and turnpike theory, Lect. Notesin Control and Information Sci. 84 (1986), 1002–1007.

19. Avdoshin S. M. e.a., Optimization of flexible production systems, Moscow Institute of Elec-

tronic Machine-building, Moscow, 1987, (preprint, in Russian).20. Dudnikov P. I. and S. N. Samborskiı, Endomorphisms of semimodules over semirings with

an idempotent operation, Preprint 87–48, Institute of Mathematics of Ukrainian Academy of

Science, Kiev, 1987, (in Russian).21. Weitzman M., Duality theory for infinite horizon convex models, Management Science 19

(1973), no. 7, 783–789.

22. Clarke F. H. and Vinter R. B., The relationship between the maximum principle and dynamicprogramming, SIAM Journal on Control and Optimization 25 (1987), no. 5, 1291–1311.

23. Denardo E., Contraction mapping in teh theory underlying dynamic programming, SIAM

Review 9 (1967), 165–177.24. McKenzie L., Turnpike theorems with technology and welfare functions variable, Mathematical

models in Economics, Academic Press, Amsterdam e.a., 1974, pp. 271–278.

25. Haurie A. Optimal control on an infinite time horizon: the turnpike approach, Journal ofMathematical Economics 3 (1976), no. 1, 81–102.

26. Peleg B. and H. Ryder, On optimal consumption plans in a multisector economy, Review ofEconomic Studies 39 (1972), no. 2, 159–169.

27. Rockafellar R. T., Saddle points of Hamiltonian systems in convex problems of Lagrange,

Journal of the Optimization Theory and Applications 12 (1973), no. 4, 367–390.28. Vorob’ev N. N., Extremal Algebra of Nonnegative Matrices, Elektronische Informationsverar-

beitung und Kybernetik 6 (1970), 303–312.

29. Leizarowitz A., Infinite Horizon Autonomous Systems with Unbounded Cost, Applied Math-ematics and Optimization 13 (1985), 19–43.

30. Gale D., On optimal development in a multi-sector economy, Review of Economic Studies 34

(1967), 1–19.31. von Weizsacker C. C., Existence of optimal programs of accumulation for an infinite horizon,

Review of Economic Studies 32 (1965), 85–104.

32. Leizarowitz A., Existence of overtaking optimal trajectories for problems with convex inte-grands, Mathematics of Operations Research 10 (1985), no. 3, 450–461.

33. Dudnikov P. I. and S. N. Samborskiı, Doklady Akademii Nauk SSSR 308 (1989), 780–783,(Translated in Soviet Math. Dokl. 40(1990)).

34. Rubinov A. M., Economic dynamics, Journal of Soviet Mathematics 26 (1984), no. 4.