Morfismos, Vol 10, No 1, 2006

VOLUMEN 10NÚMERO 1

ENERO A JUNIO DE 2006 ISSN: 1870-6525

Morfismos

Comunicaciones EstudiantilesDepartamento de Matematicas

Cinvestav

Editores Responsables

• Isidoro Gitler • Jesus Gonzalez

Consejo Editorial

• Luis Carrera • Samuel Gitler• Onesimo Hernandez-Lerma • Hector Jasso Fuentes

• Miguel Maldonado • Raul Quiroga Barranco• Enrique Ramırez de Arellano • Enrique Reyes

• Armando Sanchez • Martın Solis• Leticia Zarate

Editores Asociados

• Ricardo Berlanga • Emilio Lluis Puebla• Isaıas Lopez • Guillermo Pastor

• Vıctor Perez Abreu • Carlos Prieto• Carlos Renterıa • Luis Verde

Secretarias Tecnicas

• Roxana Martınez • Laura Valencia

ISSN: 1870 - 6525

Morfismos puede ser consultada electronicamente en “Revista Morfismos”en la direccion http://www.math.cinvestav.mx. Para mayores informes dirigirseal telefono 50 61 38 71.

Toda correspondencia debe ir dirigida a la Sra. Laura Valencia, Departa-mento de Matematicas del Cinvestav, Apartado Postal 14-740, Mexico, D.F.07000 o por correo electronico: [email protected].

VOLUMEN 10NÚMERO 1

ENERO A JUNIO DE 2006ISSN: 1870-6525

Informacion para Autores

El Consejo Editorial de Morfismos, Comunicaciones Estudiantiles del Departamento deMatematicas del CINVESTAV, convoca a estudiantes de licenciatura y posgrado a someterartıculos para ser publicados en esta revista bajo los siguientes lineamientos:

• Todos los artıculos seran enviados a especialistas para su arbitraje. No obstante, losartıculos seran considerados solo como versiones preliminares y por tanto pueden serpublicados en otras revistas especializadas.

• Se debe anexar junto con el nombre del autor, su nivel academico y la instituciondonde estudia o labora.

• El artıculo debe empezar con un resumen en el cual se indique de manera breve yconcisa el resultado principal que se comunicara.

• Es recomendable que los artıculos presentados esten escritos en Latex y sean enviadosa traves de un medio electronico. Los autores interesados pueden obtener el for-mato LATEX2ε utilizado por Morfismos en “Revista Morfismos” de la direccion webhttp://www.math.cinvestav.mx, o directamente en el Departamento de Matematicasdel CINVESTAV. La utilizacion de dicho formato ayudara en la pronta publicaciondel artıculo.

• Si el artıculo contiene ilustraciones o figuras, estas deberan ser presentadas de formaque se ajusten a la calidad de reproduccion de Morfismos.

• Los autores recibiran un total de 15 sobretiros por cada artıculo publicado.

• Los artıculos deben ser dirigidos a la Sra. Laura Valencia, Departamento de Matemati-cas del Cinvestav, Apartado Postal 14 - 740, Mexico, D.F. 07000, o a la direccion decorreo electronico [email protected]

Author Information

Morfismos, the student journal of the Mathematics Department of the Cinvestav, invitesundergraduate and graduate students to submit manuscripts to be published under thefollowing guidelines:

• All manuscripts will be refereed by specialists. However, accepted papers will beconsidered to be “preliminary versions” in that authors may republish their papers inother journals, in the same or similar form.

• In addition to his/her affiliation, the author must state his/her academic status (stu-dent, professor,...).

• Each manuscript should begin with an abstract summarizing the main results.

• Morfismos encourages electronically submitted manuscripts prepared in Latex. Au-thors may retrieve the LATEX2ε macros used for Morfismos through the web sitehttp://www.math.cinvestav.mx, at “Revista Morfismos”, or by direct request to theMathematics Department of Cinvestav. The use of these macros will help in theproduction process and also to minimize publishing costs.

• All illustrations must be of professional quality.

• 15 offprints of each article will be provided free of charge.

• Manuscripts submitted for publication should be sent to Mrs. Laura Valencia, De-partamento de Matematicas del Cinvestav, Apartado Postal 14 - 740, Mexico, D.F.07000, or to the e-mail address: [email protected]

Lineamientos Editoriales

“Morfismos” es la revista semestral de los estudiantes del Departamento de Mate-maticas del CINVESTAV, que tiene entre sus principales objetivos el que los estu-diantes adquieran experiencia en la escritura de resultados matematicos.

La publicacion de trabajos no estara restringida a estudiantes del CINVESTAV;deseamos fomentar tambien la participacion de estudiantes en Mexico y en el extran-jero, ası como la contribucion por invitacion de investigadores.

Los reportes de investigacion matematica o resumenes de tesis de licenciatura,maestrıa o doctorado pueden ser publicados en Morfismos. Los artıculos que apare-ceran seran originales, ya sea en los resultados o en los metodos. Para juzgar esto,el Consejo Editorial designara revisores de reconocido prestigio y con experiencia enla comunicacion clara de ideas y conceptos matematicos.

Aunque Morfismos es una revista con arbitraje, los trabajos se conside-raran como versiones preliminares que luego podran aparecer publicadosen otras revistas especializadas.

Si tienes alguna sugerencia sobre la revista hazlo saber a los editores y con gustoestudiaremos la posibilidad de implementarla. Esperamos que esta publicacion pro-picie, como una primera experiencia, el desarrollo de un estilo correcto de escribirmatematicas.

Morfismos

Editorial Guidelines

“Morfismos” is the journal of the students of the Mathematics Department ofCINVESTAV. One of its main objectives is for students to acquire experience inwriting mathematics. Morfismos appears twice a year.

Publication of papers is not restricted to students of CINVESTAV; we want toencourage students in Mexico and abroad to submit papers. Mathematics researchreports or summaries of bachelor, master and Ph.D. theses will be considered forpublication, as well as invited contributed papers by researchers. Papers submittedshould be original, either in the results or in the methods. The Editors will assignas referees well–established mathematicians.

Even though Morfismos is a refereed journal, the papers will be con-sidered as preliminary versions which could later appear in other mathe-matical journals.

If you have any suggestions about the journal, let the Editors know and we willgladly study the possibility of implementing them. We expect this journal to foster, asa preliminary experience, the development of a correct style of writing mathematics.

Morfismos

Contenido

A unified approach to continuous-time discounted Markov control processes

Tomas Prieto-Rumeau and Onesimo Hernandez-Lerma . . . . . . . . . . . . . . . . . . . 1

On bounds for the stability number of graphs

Isidoro Gitler and Carlos E. Valencia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Morfismos, Vol. 10, No. 1, 2006, pp. 1–40

A unified approach to continuous-timediscounted Markov control processes ∗

amreL-zednanreHomisenOuaemuR-oteirPsamoT

Abstract

In this paper we consider continuous-time Markov control pro-cesses with Borel state and action spaces. The performance crite-rion is the expected discounted reward over a finite or an infinitetime horizon. The reward rates and the transition rates of thesystem are allowed to be unbounded. We propose conditions en-suring that the optimal discounted reward of both the finite andthe infinite horizon problem satisfy a dynamic programming opti-mality equation, and we also prove the existence of ε-optimal andoptimal strategies. Finite horizon approximations to the infinitehorizon problem are discussed. We illustrate our results by show-ing that our hypotheses are satisfied by some classes of controlledMarkov chains and controlled diffusions.

2000 Mathematics Subject Classification: 93E20, 60J25, 90C40.Keywords and phrases: continuous-time Markov control processes, dis-counted optimality criterion, value iteration.

1 Introduction

This paper is concerned with continuous-time Markov control processeswith values in a Borel space, and with Borel action spaces. The criterionto be maximized is the expected discounted reward over a time horizon,which may be either finite or infinite. The transition rates of the systemand the reward rates may be both unbounded.

Many particular cases of such continuous-time Markov control pro-cesses have been extensively studied in the previous literature. We may

∗Invited Article. Research partially supported by CONACyT Grant 45693-F.

1

2 Tomas Prieto-Rumeau and Onesimo Hernandez Lerma

mention, for instance, controlled Markov chains [13], controlled jumpprocesses [12, 22], piecewise deterministic controlled processes [1, 5] orcontrolled diffusions [10, 11, 18], among others. The usual approachto deal with such problems is by means of the dynamic programmingequation or, in other words, by deriving a so-called optimality equation(or Hamilton-Jacobi-Bellman equation) that characterizes the optimalexpected reward of the control problem. Then, from this optimalityequation one can obtain the existence of “nearly” optimal strategies oreven the existence of optimal strategies.

However, in most of the existing works, the optimality equation isderived using a technique that is specific to the model under study.For instance, for controlled Markov chains, an explicit expression forthe transition probabilities is obtained in [13]; for piecewise determin-istic processes, a discrete-time embedded Markov process (determinedby the jump times) is studied in [1]; for controlled diffusions, the prop-erties of the associated second order differential operator are exploited(see [18]). Only a few references [6, 14, 15] analyze the discounted opti-mality equation for general Markov processes. Moreover, in [14, 15] onlysufficient conditions are given. More precisely, it is shown that if a givenfunction satisfies a certain optimality equation, then this function is theoptimal expected reward. But no (necessary) results proving that theoptimal expected discounted reward satisfies the optimality equationare given. For the infinite horizon case, Doshi [6] proposes a set of suchnecessary conditions for a model with bounded transition rates (that is,the generator of the process is defined only for bounded functions) andbounded reward rates.

In this paper we place ourselves in the framework of general Markovcontrol processes, and one of our goals is to generalize the results in [6]to a model with both unbounded transition and reward rates. To thisend, we impose some Lyapunov or drift conditions on the generator ofthe process yielding useful ergodic properties. Note also that Doshi [6]proposes a policy improvement algorithm whose convergence is provedunder a very severe condition, namely that the action space is finite. Inthis paper, rather than dealing with policy iteration, we study the valueiteration algorithm, that is, we study finite horizon approximations tothe infinite horizon control problem, and prove its convergence underhypotheses less restrictive than Doshi’s.

Therefore, in short, one of the main contributions of this paperis to propose fairly general assumptions that cover all of the previ-ously studied models, and so its applicability ranges from models of the

Continuous-time Markov control processes 3

continuous-state type (e.g., diffusions) to models of the discontinuoustype (e.g., Markov chains). Indeed, we illustrate our results by show-ing that our assumptions are implied by the standard ones imposed oncontrolled diffusion processes and on controlled Markov chains.

To give a flavor of the general assumptions we make, let us mentionthat they are, roughly speaking, continuity and stability assumptions.The starting point is the well-known fact that a strategy that is optimalover a time horizon, say [0,m], is necessarily optimal over all time hori-zons I ⊆ [0,m]. (Note that this is a particular feature of the discountedreward criterion but not, for example, of the average reward criterion.For a general approach to average reward Markov control processes, theinterested reader is referred to [7].)

It is worth mentioning that other approaches to continuous-timeMarkov control processes have been proposed in the literature. Theseinclude time-discretization procedures, in which the continuous-timemodel is seen as the limit of discrete-time models [1, 2]. The continuous-time optimality equation is then derived from the discrete-time opti-mality equations. In this paper, we have preferred to follow a purelycontinuous-time approach.

A linear programming formulation has also been proposed; see [16]for discrete-time models and [4, 19, 25] for their continuous-time coun-terpart. The idea is to see the optimal expected reward as being themaximum of a linear programming problem over a suitably defined spaceof occupation measures. Optimal controls are then characterized asthose reaching this maximum. Our approach is essentially different be-cause the linear programming formulation does not derive an optimalityequation, as ours does. Besides, optimal controls are defined in a weakersense. More precisely, they are optimal controls for almost every initialstate (see [25, Section 5]), whereas the approach in this paper yieldsoptimality for every initial state.

When dealing with general Markov processes, a key issue is that ofthe formal definition of these Markov processes. They may be definedstarting from the transition probability function and then deriving theKolmogorov differential equations [9, 13]. They are also frequently de-fined using the so-called generator of the process, which is a sort ofderivative of a semi-group operator [6, 14, 15]. In this paper, we havechosen the approach of the martingale characterization of the generatorof the Markov process [19, 24]. Further details are given in Remark 2.1in the next section.

The rest of the paper is organized as follows. In Section 2 we intro-


duce the continuous-time control model and give some basic definitions.Section 3 studies the finite horizon discounted problem. In Section 4we analyze the infinite horizon problem and the value iteration approx-imations. Also in this section we compare our hypotheses to those in[6]. For clarity of exposition, the proofs of the results stated in Sections3 and 4 are postponed to a later section. In Section 5 we illustrateour results by studying two particular cases: finite horizon controlleddiffusions and infinite horizon controlled Markov chains. The rest ofthe material is technical. Section 6 proves several useful lemmas. InSection 7 we prove the results stated in Sections 3 and 4. Finally, inSection 8 we provide the proofs of the results in the examples section,Section 5.

2 The control model

In this section we briefly introduce the control model. Additional spec-ifications are introduced in later sections, as needed.

The state space is X and the action space is A. They are bothassumed to be Borel spaces endowed with their respective Borel σ-algebras B(X) and B(A). For every x ∈ X, the set of admissibleactions at x is the nonempty σ-compact Borel set A(x) ⊆ A. LetK := (x, a) ∈ X × A : a ∈ A(x), assumed to be a measurable subsetof the Borel space X × A. We also consider a measure space (Ω,F).

Multifunctions. Now we recall some terminology on multifunctionsand measurable selectors (see [3], [16, Appendix D] or [23]). A set-valued function Ψ : X → 2A, where Ψ(x) = ∅ for every x ∈ X, is calleda multifunction from X to A. Let Ψ−(B) := x ∈ X : Ψ(x) ∩ B = ∅for B ⊆ A. If Ψ−(B) is closed for every closed set B ⊆ A then Ψ issaid to be upper semicontinuous. If Ψ−(B) is open for every open setB ⊆ A then Ψ is said to be lower semicontinuous. The multifunctionΨ is continuous if it is both upper and lower semicontinuous. Finally,we say that Ψ is compact-valued if A(x) is compact for every x ∈ X. Ameasurable function f : X → A such that f(x) ∈ Ψ(x) for each x ∈ Xis called a measurable selector of the multifunction Ψ.

Strategies. Loosely speaking, a Markov strategy ϕ prescribes the ac-tion ϕ(t, x) ∈ A(x) to be chosen when the state of the system at timet ≥ 0 is x ∈ X. More precisely, a Markov strategy ϕ is a measurablefunction ϕ : [0,∞) × X → A, where ϕ(t, x) ∈ A(x) for every t ≥ 0


and x ∈ X, such that for every s ≥ 0 and every initial state x ∈ Xat time s there exists a probability measure on (Ω,F) denoted Pϕ,s,x,with corresponding expectation operator Eϕ,s,x, and a right-continuousMarkov process x(t,ϕ)t≥s, with x(s,ϕ) = x, satisfying that

v(x(t,ϕ)) − v(x) −! t

s(Lϕ(u,x(u,ϕ))v)(x(u, ϕ))du for t ≥ s

is a Pϕ,s,x-martingale for every measurable v ∈ Dϕ, where the operatorL ≡ (Lav)(x)(x,a)∈ is the generator of the stochastic control process,and Dϕ is the domain of the generator of the process x(·,ϕ). This isthe martingale characterization of the generator [19, 24]. If s = 0 thenPϕ,s,x and Eϕ,s,x will be denoted by Pϕ,x and Eϕ,x, respectively.

The family of Markov strategies is denoted by Φ. If ϕ is a Markovstrategy such that ϕ(t, x) = f(x) for some measurable function f : X →A and all t ≥ 0 and x ∈ X, then we say that ϕ is a stationary strategy.In this case we also identify ϕ with the time-independent function f .The set of stationary strategies is denoted by F. Observe that ϕ ∈ Φ isa measurable selector for the multifunction from [0,∞)×X to A definedby (s, x) (→ A(x), and, similarly, f ∈ F is a measurable selector for themultifunction from X to A defined by x (→ A(x).

Remark 2.1 The above definition of the Markov control process mayseem vague: indeed, the existence of the corresponding probability mea-sure Pϕ,s,x is assumed yet not proved. Note, however, that in the papers[4, 19, 24], whose authors also propose the martingale characterizationas definition of the Markov process, the existence of a stationary Markovcontrol process is established. Proving the existence of a nonstationaryMarkov control process is beyond the scope of this paper. In spite ofthis, for a given model, the existence of such nonstationary controls canbe proved by imposing some specific assumptions. As an illustration,this is the case for controlled diffusions [18], where some regularity as-sumptions on the parameters of the stochastic differential equations aremade, and for controlled Markov chains [9, 13], where the continuity ofthe transition rates is assumed.

Sometimes we will also need to use the so-called extended generatorof the process x(·,ϕ), where ϕ ∈ Φ. Whereas the generator of theprocess is defined for functions with domain X, the extended generatoris defined for “time-space” functions. Given a measurable function v :


[0,∞) × X → R, we say that v belongs to the domain of the extendedgenerator of the process x(·,ϕ), denoted by v ∈ Dϕ, if

v(t, x(t,ϕ)) − v(s, x) −! t

s(Lϕ(u,x(u,ϕ))v)(u, x(u,ϕ))du(1)

for t ≥ s is a Pϕ,s,x-martingale for every s ≥ 0 and every initial statex ∈ X at time s. Observe that we use the notation L for both thegenerator and the extended generator.

Suppose that the reward rate is the Borel measurable function r :K → R and that a discount factor α > 0 is given. In this paper we shalldeal with both finite and infinite horizon problems.

Finite horizon problems. Consider a finite horizon control problem,where the time horizon is [0,m], for m > 0. Given ϕ ∈ Φ, 0 ≤ s ≤ mand an initial state x ∈ X at time s, define the expected discountedreward of ϕ on [s,m] as

V m(ϕ, s, x) := Eϕ,s,x

! m

se−α(t−s)r(x(t,ϕ),ϕ(x(t,ϕ)))dt.(2)

The optimal reward of the control problem over the time interval [s,m]for a given initial state x ∈ X at time s is then defined as

V ∗m(s, x) := supϕ∈Φ

V m(ϕ, s, x),(3)

assumed to be measurable (this measurability assumption is not restric-tive in practice). We say that ϕ ∈ Φ is optimal for the control problemwith finite horizon [0,m] if

V m(ϕ, s, x) = V ∗m(s, x) for every x ∈ X and 0 ≤ s ≤ m,

and that ϕ ∈ Φ is ε-optimal, where ε > 0, for the control problem withfinite horizon [0,m] if

V m(ϕ, s, x) + ε ≥ V ∗m(s, x) for every x ∈ X and 0 ≤ s ≤ m.

Clearly, when dealing with finite horizon problems, the extendedgenerator needs to be defined only for functions v : [0,m]×X → R, andthen the martingale property (1) is only required for s ≤ t ≤ m.


Infinite horizon problems. For infinite horizon problems, given aMarkov strategy ϕ ∈ Φ, define the (infinite horizon) expected discountedreward of ϕ when the initial state at time s ≥ 0 is x ∈ X as

V (ϕ, s, x) := Eϕ,s,x

! ∞

se−α(t−s)r(x(t,ϕ),ϕ(t, x(t,ϕ)))dt.(4)

If s = 0 we simply write V (ϕ, 0, x) =: V (ϕ, x). The optimal expecteddiscounted reward is given by

V ∗(s, x) := supϕ∈Φ

V (ϕ, s, x),(5)

which for s = 0 reduces to

V ∗(x) := supϕ∈Φ

V (ϕ, x),(6)

assumed to be measurable. In fact, it will be shown in Lemma 6.1 belowthat V ∗(s, x) = V ∗(x) for every x ∈ X and s ≥ 0. A Markov strategyϕ ∈ Φ is optimal for the infinite horizon problem if V (ϕ, x) = V ∗(x) forevery x ∈ X, and it is said to be ε-optimal, for ε > 0, if V (ϕ, x) + ε ≥V ∗(x) for every x ∈ X.

Our assumptions will ensure that the above defined expressions, i.e.,(2)–(6), are well defined and finite.

Consider a given measurable function W : X → [1,∞) such thatthe level sets x ∈ X : W (x) ≤ C have compact closure for everyC ≥ 1. Such a function is called a Lyapunov function; it is also knownas a moment or a norm-like function. Let BW be the Banach space ofmeasurable functions v : X → R with norm

||v||W := supx∈X

|v(x)|W (x)

< ∞.

We will make the following assumption on the control model.

Assumption A.

(a) The function W is in Dϕ for every ϕ ∈ Φ, and there exist constants0 < c < α and b ≥ 0 such that

(LaW )(x) ≤ cW (x) + b for every (x, a) ∈ K.

(b) There exists a constant M > 0 such that

|r(x, a)| ≤ MW (x) for each (x, a) ∈ K.


Assumption A(a) imposes a so-called Lyapunov or drift conditionon the generator. A similar assumption is made in the papers [12, 13,20], among others. This is a standard assumption when dealing withunbounded cost or reward rates because it imposes a growth conditionon the reward rate; see Lemma 6.2 below and also, for instance, [17,Chapter 8] or [12, 13].

It is worth noting that the positivity of the constant c in AssumptionA(a) is not strictly necessary. Indeed, if c < 0 then the bound obtainedin Lemma 6.2 below should be modified in accordance (see [13, Theorem3.1]) and our results would still be valid.

3 The finite horizon case

In this section we study the optimality equation for finite horizon prob-lems, as well as the existence of ε-optimal or optimal strategies.

First, we introduce some notation. Fix x ∈ X, a neighborhood Nof x, and s ≥ 0. Given ϕ ∈ Φ, consider the Markov process x(t,ϕ)t≥s

with initial state x at time s. Define τ(x,N, ϕ, s) as the exit time of theprocess x(·, ϕ) from N , that is,

τ(x,N, ϕ, s) := inft ≥ s : x(t,ϕ) /∈ N.

If s = 0 we will simply write

τ(x, N, ϕ) := τ(x,N, ϕ, 0).(7)

Our next assumption imposes some continuity and stability conditions.Parts (a)–(c) in Assumption B below are minimal continuity require-ments, whereas part (d) excludes “instantaneous jumps” of the stateprocess; in other words, for each initial state x at time s, the state pro-cess x(·,ϕ) remains in a neighborhood N of x a positive time period,uniformly in ϕ ∈ Φ. We will provide in Lemma 6.8 an easily verifiablesufficient condition for Assumption B(d).

Assumption B.

(a) For each m > 0 the function (s, x) #→ V ∗m(s, x) is continuous on[0,m] × X and it belongs to Dϕ for every ϕ ∈ Φ.

(b) For each m > 0 the function

(s, x) #→ supa∈A(x)

r(x, a) + (LaV ∗m)(s, x)


is continuous on [0,m] × X.

(c) The function a "→ r(x, a) + (LaV ∗m)(s, x) is continuous on A(x)for every fixed 0 ≤ s ≤ m and x ∈ X.

(d) Given x ∈ X, a neighborhood N of x, s ≥ 0 and δ > 0,

supϕ∈Φ

Eϕ,s,x[exp−α minτ(x,N,ϕ, s) − s, δ] < 1.

Concerning Assumption B(d), see Lemma 6.8.The next result is similar to [14, Theorem 5.1] or [15, Theorem 6.1].

Observe however that in these references the existence of a solution tothe optimality equations (8)–(9) is assumed, whereas in Theorem 3.1 weprove the existence of this solution.

Theorem 3.1 Consider a control problem with finite horizon [0,m]. IfAssumptions A and B are verified, then V ∗m is the unique functionv : [0,m] × X → R in ∩ϕ∈ΦDϕ satisfying Assumption B(c) that is asolution of the equation

αv(s, x) = supa∈A(x)

r(x, a) + (Lav)(s, x) for 0 ≤ s < m, x ∈ X,(8)

with terminal condition

v(m,x) = 0 for x ∈ X.(9)

Moreover, for each given ε > 0, there exist ε-optimal strategies for thefinite horizon problem on [0,m].

The following two corollaries propose conditions under which thereexist optimal strategies. This is achieved requiring that either Assump-tions A and B hold and that A(x) is compact for every x ∈ X, or relaxingAssumptions B(b) and B(c) and strengthening instead the hypotheseson the multifunction x "→ A(x).

Corollary 3.2 Suppose that Assumptions A and B are verified and thatthe action set A(x) is compact for every x ∈ X. Then the conclusions ofTheorem 3.1 remain valid and, moreover, there exist optimal strategiesfor the finite horizon problem.


Corollary 3.3 Suppose that Assumptions A, B(a), and B(d) are veri-fied and, in addition, the function

(s, x, a) !→ r(x, a) + (LaV ∗m)(s, x)

is continuous on [0,m]×K. Assume further that the multifunction fromX to A defined by x !→ A(x) is continuous and compact-valued. Thenthe conclusions of Theorem 3.1 remain valid and, moreover, there existoptimal strategies for the finite horizon problem.

4 The infinite horizon problem

In this section we study the infinite horizon control problem: the op-timality equation and the existence of ε-optimal or optimal strategies.Actually we consider two different approaches.

First, we consider finite horizon approximations — which can bealso described as a successive approximations approach — to the infi-nite horizon problem, that is, we explore the limiting behavior of theresults in the previous section as the time horizon goes to infinity. Thisapproach requires restrictive assumptions (see Assumptions C and Dbelow) but it could be extremely helpful for practical purposes because,as shown in Theorem 4.1, the convergence of V ∗m(0, ·) to V ∗ is exponen-tial and, furthermore, given ε > 0, we can explicitly determine a timehorizon m such that ||V ∗m(0, ·) − V ∗||W ≤ ε (see Remark 4.2).

In our second approach, under reasonably mild hypotheses (see As-sumption E) we obtain the optimality equation and the existence ofoptimal policies for the infinite horizon problem but, unfortunately, weget no information on the finite horizon approximations.

It should be noted that our results in this section prove the existenceof a solution to the optimality equation, whereas in previous papers (e.g.,[14, Theorem 6.2] or [15, Theorem 4.1]) the existence of such a solutionwas assumed.

The successive approximations approach. We will require thefollowing assumption. (Recall the notation in (4).)

Assumption C. For every ϕ ∈ Φ, V (ϕ, ·, ·) is in Dϕ and, furthermore,the functions

(s, x) !→ V (ϕ, s, x)and

(s, x) !→ r(x,ϕ(s, x)) + (Lϕ(s,x)V (ϕ, ·, ·))(s, x)are continuous on [0,∞) × X.


Theorem 4.1 below deals with the value iteration approximations,i.e., the convergence of V ∗m to V ∗ as m → ∞. In the literature ondiscrete-time control problems, it is well known that, under suitableconditions, the value iteration procedure converges geometrically, thatis, V ∗m converges at a geometric rate to V ∗. The reason for this isthat V ∗ is the fixed point of a contraction mapping; see, for instance[17, Theorem 8.3.6]. Theorem 4.1 shows that this result is also true forcontinuous-time problems.

Theorem 4.1 If Assumptions A, B and C are verified, then, for everyT > 0, V ∗m(s, ·) converges exponentially to V ∗ uniformly on 0 ≤ s ≤ Tin the W -norm as m → ∞; in symbols,

sup0≤s≤T

||V ∗m(s, ·) − V ∗(·)||W = O(e−(α−c)m) as m → ∞.

Remark 4.2 In the proof of Theorem 4.1, we will derive an explicitexpression for the term O(e−(α−c)m) above depending only on knownconstants: b, c, α, M and T (see (38)). Therefore, for a given ε > 0, wecan effectively determine a value of m for which sup0≤s≤T ||V ∗m(s, ·) −V ∗(·)||W ≤ ε.

As a consequence of Theorem 4.1, we obtain a sufficient conditionfor the continuity of V ∗, which is sometimes required (see AssumptionE(b) below).

Corollary 4.3 Suppose that the state space X is locally compact andthat the Lyapunov function W is bounded on compact sets. If Assump-tions A, B and C are satisfied, then V ∗ is continuous on X.

Theorem 4.1, together with the finite horizon optimality equation inTheorem 3.1, i.e.,

αV ∗m(s, x) = supa∈A(x)

r(x, a) + (LaV ∗m)(s, x) for 0 ≤ s < m, x ∈ X,

suggests that letting m → ∞ in this equation would yield the infinitehorizon optimality equation (10) below with v = V ∗. To this end, wemust impose the following condition.

We consider the family of functions on [0, 1] × X

(s, x) '→ Gf,m(s, x) := (Lf(x)V ∗m)(s, x) − (Lf(x)V ∗)(x),

parametrized by f ∈ F and the time horizon m ∈ N.


Assumption D.

(a) The infinite horizon optimal reward V ∗ is in Dϕ for every ϕ ∈ Φ.

(b) The family of functions Gf,mf∈ ,m∈ is equicontinuous at (0, x)for every x ∈ X.

To prove the convergence of the policy iteration algorithm, a hypoth-esis similar to the equicontinuity condition Assumption D(b) is made in[6, Assumption 8].

Theorem 4.4 If Assumptions A, B, C and D are satisfied, then V ∗ isthe unique function v in (

!ϕ∈Φ Dϕ) ∩ BW satisfying Assumption B(c)

that is a solution of the optimality equation

αv(x) = supa∈A(x)

r(x, a) + (Lav)(x) for x ∈ X.(10)

Moreover, for every ε > 0 there exist ε-optimal stationary strategies.

In connection with this theorem see Corollary 4.6.

Direct approach to the infinite horizon problem. So far we haveanalyzed the successive approximations approach to the infinite horizonproblem. Now, we drop Assumptions C and D and, instead, we proposea less restrictive assumption (Assumption E below) allowing us to derivedirectly the infinite horizon optimality equation, but we do not obtaininformation on the finite horizon approximations. (In Assumption E(d),recall the notation in (7).)

Assumption E.

(a) The function V ∗ is in ∩ϕ∈ΦDϕ.

(b) The functions V ∗ and x #→ supa∈A(x)r(x, a)+(LaV ∗)(x) are con-tinuous on X.

(c) For each x ∈ X, a #→ r(x, a) + (LaV ∗)(x) is continuous on A(x).

(d) Given x ∈ X and a neighborhood N of x

supϕ∈Φ

Eϕ,x[e−ατ(x,N,ϕ)] < 1.


Observe that Assumption B(d) —and hence the condition in Lem-ma 6.8— implies Assumption E(d).

The following theorem gives the same conclusions as Theorem 4.4although, for clarity of exposition, we prefer to state it separately.

Theorem 4.5 Suppose that Assumptions A and E hold. Then V ∗ isthe unique function v in (

!ϕ∈Φ Dϕ) ∩ BW satisfying Assumption E(c)

that is a solution of the optimality equation


r(x, a) + (Lav)(x) for x ∈ X.

Moreover, for every ε > 0 there exist ε-optimal stationary strategies.

The following corollaries are similar to Corollaries 3.2 and 3.3 on thefinite-horizon problem.

Corollary 4.6 Suppose that either Assumptions A, B, C and D, orAssumptions A and E are verified. Suppose also that A(x) is compactfor every x ∈ X. Then the conclusions of Theorems 4.4 and 4.5 remainvalid and, moreover, there exist optimal stationary strategies.

Corollary 4.7 Suppose that Assumptions A, E(a), and E(d) are veri-fied and that the functions

x #→ V ∗(x) and (x, a) #→ r(x, a) + (LaV ∗)(x)

are continuous on X and K, respectively. Assume also that the multi-function from X to A defined by x #→ A(x) is continuous and compact-valued. Then the conclusions of Theorem 4.5 remain valid and, further-more, there exist optimal stationary strategies.

To conclude this section, let us compare our assumptions to thosein Doshi [6]. The proof techniques are essentially the same, combiningcontinuity and stability features. There are however two important im-provements. The first one is that we allow the reward rates and thetransition rates (i.e., the generator) to be unbounded; see [6, Definition2.4] and [6, Assumption 1(d)]. Second, our stability condition (for in-stance, Assumption E(d)) is, by far, more general than the correspond-ing stability condition stated in Assumption 5 in [6, p. 1226]. Indeed, asshown in Lemma 6.8 below, a sufficient condition for Assumption E(d)


is that given x ∈ X, a neighborhood N of x and s ≥ 0, there existsδ0 > 0 such that

infϕ∈Φ

Pϕ,s,xτ(x,N, ϕ, s) ≥ s + δ0 > 0,(11)

whereas [6, Assumption 5] would require the existence of δ0 > 0 with

infϕ∈Φ

Pϕ,s,xτ(x,N, ϕ, s) ≥ s + δ0

arbitrarily close to one, which of course is much more restrictivethan (11). The reason for this is that to prove a result such as Theo-rem 3.1, for instance, Doshi considers deterministic intervals, say [s, s+δ0], while in our proofs we deal with random intervals [s, τ(x,N,ϕ, s)].

5 Examples

Finite horizon controlled diffusions. The topic of controlled diffu-sions over finite intervals has been extensively studied by many authors,in particular, by Krylov [18]. We are going to show that his hypothe-ses on the control model imply our assumptions and, therefore, we canuse the results in Section 3 to prove the existence of a solution to theoptimality equation and the existence of optimal controls.

For expositional ease, we will analyze one-dimensional controlled dif-fusions, though the results may be generalized to the multidimensionalcase.

Suppose that the state space is R and that the set of admissiblecontrols is a compact Borel space A. We fix a time horizon [0, T ], whereT > 0. Let Wt,Ftt≥0 be a one-dimensional Brownian motion.

We consider real-valued functions b(t, x, a) and σ(t, x, a), defined for0 ≤ t ≤ T , x ∈ R and a ∈ A. Suppose that the state of the systemis x at time t ∈ [0, T ) and that we use the control a ∈ A. Then, theinfinitesimal evolution of the state of the system is determined by

dx = b(t, x, a)dt + σ(t, x, a)dWt.

This formula will be given a formal definition in the next paragraph.Define Ct as the family of continuous functions from [0, t] to R.

Let Nt, for t ≥ 0, be the minimal σ-algebra that contains the sets ofthe form x ∈ Ct : x(s) ∈ Γ for s ∈ [0, t] and Γ ∈ B, the Borel σ-algebra of R. We say that ϕ := ϕ(t, ·)0≤t≤T is an admissible strategy


if ϕ(t, ·) : Ct → A is Nt-measurable for all 0 ≤ t ≤ T and, moreover, forevery s ∈ [0, T ] and x ∈ R, there exists a solution of

x(t) = x +! t

0b(s + u,x(u),ϕ(u,x))du +

! t

0σ(s + u,x(u), ϕ(u,x))dWu

for 0 ≤ t ≤ T − s, which is adapted to Ftt≥0. Then we definex(t, ϕ) := x(t−s) for s ≤ t ≤ T , which has initial value x(s,ϕ) = x. As-sumption CD below ensures the existence of such solutions. The proba-bility measure Pϕ,s,x and the expectation operator Eϕ,s,x correspondingto x(t,ϕ)s≤t≤T are given the same definitions as in Section 2.

It follows that history-dependent strategies are also admissible. Wedenote by Φ the set of admissible strategies, which contains Φ, the familyof Markov strategies. We define V T (ϕ, s, x) as in (2), that is

V T (ϕ, s, x) := Eϕ,s,x

! T

se−α(t−s)r(x(t,ϕ),ϕ(t, x(t,ϕ)))dt,(12)

(recall that r is the reward rate) and (cf. (3))

V ∗T (s, x) := supϕ∈Φ

V T (ϕ, s, x)(13)

for s ∈ [0, T ] and x ∈ R.We impose the following Assumption CD (where CD stands for “con-

trolled diffusions”), which is drawn from [18].

Assumption CD.

(a) The functions b(t, x, a), σ(t, x, a) and r(x, a) are continuous withrespect to (t, x, a) ∈ [0, T ] × R × A, uniformly in a ∈ A.

(b) There exist nonnegative constants K and p such that

|σ(t, x, a) − σ(t, y, a)| + |b(t, x, a) − b(t, y, a)| ≤ K|x − y|,

|σ(t, x, a)|+ |b(t, x, a)| ≤ K(1+ |x|) and |r(x, a)| ≤ K(1+ |x|)p

for all 0 ≤ t ≤ T , a ∈ A and x, y ∈ R.

(c) The functions b, σ and r have derivatives with respect to t ∈ [0, T ]and second derivatives with respect to x ∈ R, for each a ∈ A.Assume also that these derivatives are continuous on [0, T ] × Rand that they are bounded by K(1 + |x|)p.


(d) For each R > 0 there exists some δR > 0 such that

|σ(t, x, a)| ≥ δR

for t ∈ [0, T ], a ∈ A and |x| ≤ R.

These assumptions are quite standard in the theory of stochasticdifferential equations.

By Vt, Vx and Vxx we will denote the derivative with respect tot ∈ [0, T ] and the first and second derivatives with respect to x ∈ R,respectively, of a real-valued function V defined on [0, T ]×R. If Vt andVxx are continuous on [0, T ] × R we will write V ∈ C1,2([0, T ] × R).

In Theorem 3.1 we proved that Assumptions A and B implied theexistence of a solution to the finite horizon optimality equation. Noticehowever that Assumption A was used only to ensure the finiteness of(12) and (13), which in fact can be derived from Assumption CD; see thefootnote in [18, p. 131]. Therefore, it suffices to prove that AssumptionCD implies Assumption B.

Theorem 5.1 Suppose that Assumption CD holds and that V ∗Tt and

V ∗Txx are continuous on [0, T ]×R. Then Assumption B is satisfied and,

therefore, V ∗T is the unique solution v in (∩ϕ∈ΦDϕ) ∩ C1,2([0, T ] × R)of the equation

αv(t, x) = supa∈A r(x, a) + vt(t, x) + b(t, x, a)vx(t, x)

+ 12σ2(t, x, a)vxx(t, x)

!

for 0 ≤ t < T , and v(T, x) = 0, for every x ∈ R.Moreover, there exists an optimal Markov strategy.

Observe that Theorem 4.7.7 in [18] ensures the existence of V ∗Tt and

V ∗Txx but their continuity cannot be proved in such a general framework.

However, for specific models, the continuity of the derivatives can beshown; see, for instance, the one dimensional controlled diffusion withtwo boundaries analyzed in [18, Section 1.4].

Furthermore, under Assumption CD, [18, Theorem 4.7.7] shows thatV ∗T is a solution of the optimality equation almost-everywhere in [0, T ]×R. Also, under Assumption CD, a uniqueness result is proved in [18,Theorem 5.3.14].

As a conclusion, we have reached the same results as in [18] butusing a general approach.


Infinite horizon controlled Markov chains. We consider a contin-uous-time controlled Markov chain with denumerable state space. Ourapproach in based on the paper [13].

Without loss of generality, we assume that the state space X is theset of nonnegative integers. Define q(y|x, a) as the transition rate fromstate x ∈ X to state y ∈ X when the action a ∈ A(x) is chosen. Notethat we do not assume that the Borel space A(x) is σ-compact. Indeed,this hypothesis was needed to derive the existence of a measurable selec-tor as in [23, Corollary 4.3]. In our particular case, since X is discrete,any function f : X → A with f(x) ∈ A(x) is a measurable selector.

Admissible strategies ϕ are defined as those for which

t #→ qxy(t,ϕ) := q(y|x,ϕ(t, x))

is continuous on [0,∞) for every x, y ∈ X. In [13] randomized strategiesare also considered but, in fact, it is easily shown that we can restrictourselves to deterministic strategies, as we did in Section 2.

The following Assumption CMC (where CMC stands for “controlledMarkov chains”) is taken from [13].

Assumption CMC.

(a) The transition rates are conservative and stable, that is!

y∈X

q(y|x, a) = 0 and q(x) := supa∈A(x)

−q(x|x, a) < ∞

for every (x, a) ∈ K.

(b) Assumption A (see Section 2) is satisfied.

(c) For each x ∈ X, A(x) is compact and the functions r(x, a) (r is thereward rate), q(y|x, a) and

"y∈X q(y|x, a)W (y) are continuous on

A(x) for each x, y ∈ X.

(d) There exists a nonnegative function W ′ on X and constants c′ > 0,b′ ≥ 0 and M ′ > 0 such that

q(x)W (x) ≤ M ′W ′(x) and!

y∈X

q(y|x, a)W ′(y) ≤ c′W ′(x) + b′

for (x, a) ∈ K.


Observe that Assumptions A(1) and A(2) in [13] are omitted inAssumption CMC above because they are implied by the fact that Wis a Lyapunov function.

It is proved in [13] that Assumptions CMC(a) and CMC(b) implythat V ∗ is the unique solution in BW of the infinite horizon optimalityequation and that there exist ε-optimal stationary policies. Further-more, if Assumptions CMC(c) and CMC(d) are satisfied, then thereexists an optimal stationary policy.

Let us now show that our assumptions in this paper are implied byAssumption CMC.

Theorem 5.2 (i) Suppose that Assumptions CMC(a) and CMC(b) aresatisfied. Then Assumptions A and E are verified and, therefore,V ∗ is the unique solution v in BW of the equation


r(x, a) +!

y∈X

q(y|x, a)v(y) for x ∈ X

and, for every ε > 0, there exist ε-optimal stationary strategies.

(ii) If, in addition, Assumptions CMC(c)–(d) are satisfied, then thehypotheses of Corollary 4.6 are verified and, therefore, there existoptimal stationary strategies.

This theorem shows that our general approach may be used to derivethe result stated in [13, Theorem 3.2] for continuous-time controlledMarkov chains.

6 Preliminary results

In this section we present several useful results. Although most of themare well known in the literature of continuous-time control models, weinclude them here for completeness.

Lemma 6.1 For every x ∈ X and s ≥ 0

V ∗(s, x) := supϕ∈Φ

V (ϕ, s, x) = supϕ∈Φ

V (ϕ, x) =: V ∗(x).

Proof. For an arbitrary ϕ ∈ Φ define ϕ+s ∈ Φ as

ϕ+s(t, ·) := ϕ(s + t, ·) for t ≥ 0,


and ϕ−s ∈ Φ as!

ϕ−s(t, ·) := ϕ(0, ·) if 0 ≤ t < s

ϕ−s(t, ·) := ϕ(t − s, ·) if t ≥ s.

It turns out that

V (ϕ, s, ·) = V (ϕ+s, ·) and V (ϕ, ·) = V (ϕ−s, s, ·).

Taking the supremum in these equalities yields the stated fact. !

Our next result, which is stated without proof, is a consequence of[20, Theorem 2.1(iii)]. See also [13, Theorem 3.1].

Lemma 6.2 Suppose that Assumption A(a) is verified and let ϕ ∈ Φ.Then for every s ≥ 0 and x ∈ X

Eϕ,s,xW (x(t,ϕ)) ≤ ec(t−s)W (x) +b

c(ec(t−s) − 1) for every t ≥ s.

This lemma, together with Assumption A(b), shows that the expres-sions (2)–(6) are well defined and finite. This lemma also ensures thatthe functions we will deal with (e.g. V (ϕ, s, ·), V ∗, V ∗m(s, ·), etc.) arein BW .

We state without proof the standard “product derivative” propertyof the generator L (see, e.g., [14, Lemma 2.1]).

Lemma 6.3 Suppose that ϕ ∈ Φ and v : [0,∞) × X → R in Dϕ aregiven. For α > 0, define the function v : [0,∞) × X → R as v(t, x) :=e−αtv(t, x). Then v is in Dϕ and

(Lϕ(t,x)v)(t, x) = e−αt(Lϕ(t,x)v)(t, x) − αe−αtv(t, x)

for x ∈ X and t ≥ 0.

Our following three lemmas are usually known as verification theo-rems.

Lemma 6.4 Suppose that Assumption A is verified. Fix a time horizon[0,m], where m > 0. Given ϕ ∈ Φ, if a function v : [0,m] × X → R inDϕ satisfies, for all x ∈ X and 0 ≤ t < m,

αv(t, x) = r(x,ϕ(t, x)) + (Lϕ(t,x)v)(t, x),(14)v(m,x) = 0,


thenv(s, x) = V m(ϕ, s, x) for all x ∈ X and s ∈ [0, m].(15)

If (14) holds with inequality ≥ or ≤ then the equality in (15) is replacedwith ≥ or ≤, respectively.

Proof. We fix s ∈ [0,m) and an initial state x ∈ X at time s. By (14)we have, for s ≤ t ≤ m,

αv(t, x(t,ϕ)) = r(x(t,ϕ),ϕ(t, x(t,ϕ))) + (Lϕ(t,x(t,ϕ))v)(t, x(t,ϕ))(16)

v(m,x(m,ϕ)) = 0.(17)

Multiplying (16) by e−α(t−s), integrating over the interval [s,m], andthen taking expectation Eϕ,s,x gives

Eϕ,s,x!" m

s αe−α(t−s)v(t, x(t, ϕ))dt#

= V m(ϕ, s, x) + Eϕ,s,x!" m

s e−α(t−s)(Lϕ(t,x(t,ϕ))v)(t, x(t,ϕ))dt#.

(18)

Applying Lemma 6.3 to v : (t, x) $→ e−αtv(t, x), from (18) and usingDynkin’s formula, we obtain

0 = V m(ϕ, s, x) + eαs[e−αmEϕ,x,s[v(m, x(m,ϕ))] − e−αsv(s, x)],

and thus (recall (17)) v(s, x) = V m(ϕ, s, x).The proof when the equality is replaced with an inequality is made

using the same technique. !

In Lemma 6.5 below we state a verification theorem for the infinitehorizon discounted payoff of a stationary strategy. The correspondingresult for a nonstationary strategy is given in Lemma 6.6.

Lemma 6.5 Suppose that Assumption A is satisfied, and let f ∈ F.The infinite horizon discounted reward V (f, ·) is the unique solution vin Df ∩ BW of the equation

αv(x) = r(x, f(x)) + (Lf(x)v)(x) for every x ∈ X.(19)

If (19) holds with inequality ≥ or ≤, then v ≥ V (f, ·) or v ≤ V (f, ·),respectively.


Proof. Lemma 6.2 implies that the function x !→ r(x, f(x)) is in thedomain of the α-resolvent of the Markov process x(·, f). Indeed,

Ef,x

!" ∞

0e−αt|r(x(t, f), f(x(t, f)))|dt

#

≤ M

" ∞

0e−αtEf,x[W (x(t, f))]dt < ∞.

As a consequence of [8, Lemma 4.3], it follows that V (f, ·) ∈ Df is asolution of (19).

Suppose now that v ∈ Df ∩ BW satisfies (19). Then

v(x(t, f)) = r(x(t, f), f(x(t, f))) + (Lf(x(t,f))v)(x(t, f)) for all t ≥ 0.

Multiplying this expression by e−αt, integrating over [0, T ], then takingexpectation Ef,x, and using Dynkin’s formula as in Lemma 6.4, givesthat

v(x) = Ef,x

!" T

0e−αtr(x(t, f), f(x(t, f)))dt

#+ e−αTEf,x[v(x(T, f))].

By Lemma 6.2 and dominated convergence, it follows that

limT→∞

Ef,x

!" T

0e−αtr(x(t, f), f(x(t, f)))dt

#= V (f, x).

Also by Lemma 6.2 and recalling that v is in BW , e−αTEf,x[v(x(T, f))]tends to zero as T → ∞. It follows that v = V (f, ·).

The result for the inequalities is derived similarly. !

Lemma 6.6 Suppose that Assumptions A, B(d) and C are verified.Given ϕ ∈ Φ, s ≥ 0 and x ∈ X, the following holds:

αV (ϕ, s, x) = r(x,ϕ(s, x)) + (Lϕ(s,x)V (ϕ, ·, ·))(s, x).(20)

Proof. The proof technique is similar to that of Theorem 3.1 and, there-fore, we will skip some details. We proceed by contradiction. If thestated result does not hold then there exist s ≥ 0 and x ∈ X such thatfor some ε > 0 either

αV (ϕ, s, x) + 2ε ≤ r(x,ϕ(s, x)) + (Lϕ(s,x)V (ϕ, ·, ·))(s, x)(21)

orαV (ϕ, s, x) − 2ε ≥ r(x,ϕ(s, x)) + (Lϕ(s,x)V (ϕ, ·, ·))(s, x).(22)


Suppose for instance that (21) holds. By the continuity Assumption C,there exist δ > 0 and a neighborhood N of x such that

αV (ϕ, t, y) + ε ≤ r(y, ϕ(t, y)) + (Lϕ(t,y)V (ϕ, ·, ·))(t, y)(23)

for s ≤ t ≤ s + δ and y ∈ N . Define

τδ := minτ(x,N,ϕ, s), s + δ,

and notice that Assumption B(d) implies that

Eϕ,s,x[e−α(τδ−s)] < 1.

Inequality (23) gives

αV (ϕ, t, x(t,ϕ)) + ε

≤ r(x(t,ϕ), ϕ(t, x(t,ϕ))) + (Lϕ(t,x(t,ϕ))V (ϕ, ·, ·))(t, x(t,ϕ))

for s ≤ t ≤ τδ. Multiplying this expression by e−α(t−s), integratingover [s, τδ], taking expectation Eϕ,s,x, and then using Dynkin’s formulayields

ε

α(Eϕ,s,x[e−α(τδ−s)] − 1) ≥ 0,

which is a contradiction.Similary, one can prove that (22) cannot hold, and the stated result

follows. !

Observe that to prove the converse of Lemma 6.6 (cf. Lemma 6.5)and, in order to use the same technique as in the proof of Lemma 6.5, wewould need to impose some boundedness assumptions on the functionv(·, x) solution of (20).

The next result deals with the existence of measurable maximizersfor a multifunction. For its proof see Theorems 1 and 2 in [3, SectionVI.3], for instance.

Lemma 6.7 Suppose that the multifunction Ψ from X to A is contin-uous and compact-valued. Let g : K → R be a continuous function anddefine the function h as

h(x) := maxa∈Ψ(x)

g(x, a) for x ∈ X.

Then there exists a measurable selector f : X → A such that h(x) =g(x, f(x)) for each x ∈ X and, moreover, h is continuous.


Finally, to conclude this section we propose a more easily verifiablesufficient condition for Assumption B(d).

Lemma 6.8 Suppose that given x ∈ X, a neighborhood N of x, ands ≥ 0, there exists δ0 > 0 such that

infϕ∈Φ

Pϕ,s,xτ(x,N, ϕ, s) ≥ s + δ0 > 0.

Then Assumption B(d) is verified.

Proof. Fix x ∈ X, a neighborhood N of x and s ≥ 0. Observe that thefunction

δ #→ infϕ∈Φ

Pϕ,s,xτ(x,N, ϕ, s) ≥ s + δ

is decreasing, and so, our hypothesis implies that

infϕ∈Φ

Pϕ,s,xτ(x,N, ϕ, s) ≥ s + δ > 0 for all 0 < δ ≤ δ0.

Now fix ϕ ∈ Φ and, to simplify the notation, let

D(δ,ϕ) := τ(x,N, ϕ, s) ≥ s + δ.

Note that Eϕ,s,x[exp−α minτ(x,N,ϕ, s) − s, δ] equals

Eϕ,s,x[ID(δ,ϕ)e−αδ] + Eϕ,s,x[IDc(δ,ϕ) exp−α(τ(x,N,ϕ, s) − s)].

As a consequence,

Eϕ,s,x[exp−α minτ(x,N, ϕ, s)−s, δ] ≤ 1−(1−e−αδ)Pϕ,s,x(D(δ,ϕ)).

Hence,

supϕ∈Φ

Eϕ,s,x[exp−α minτ(x,N, ϕ, s) − s, δ] < 1

for all 0 < δ ≤ δ0. Observe now that

δ #→ supϕ∈Φ

Eϕ,s,x[exp−α minτ(x,N, ϕ, s) − s, δ]

is a decreasing function, from which we conclude that Assumption B(d)is satisfied. !


7 Proof of results in Sections 3 and 4

Proof of Theorem 3.1. It is obvious that V ∗m satisfies (9). To prove(8) we will proceed as follows.

(24)

(i) Show that

αV ∗m(s, x) ≤ supa∈A(x)

r(x, a) + (LaV ∗m)(s, x)

for 0 ≤ s < m and x ∈ X.

(ii) Prove that there exist ε-optimal strategies.

(iii) Show that

αV ∗m(s, x) ≥ supa∈A(x)

r(x, a) + (LaV ∗m)(s, x)

for 0 ≤ s < m and x ∈ X.

(iv) Finally, we will establish that V ∗m is the unique solution of (8)–(9)that satisfies Assumption B(c).

Proof of (i). We proceed by contradiction. If (24) does not hold thenthere exist 0 ≤ s < m and x ∈ X such that

V ∗m(s, x) > supa∈A(x)

r(x, a) + (LaV ∗m)(s, x).

Our continuity assumptions B(a) and B(b) ensure that there exist ε > 0,δ ∈ (0, m − s) and a neighborhood N of x such that for every y ∈ Nand t ∈ [s, s + δ]

V ∗m(t, y) − ε > supa∈A(y)

r(y, a) + (LaV ∗m)(t, y),

and thus for every ϕ ∈ Φ

V ∗m(t, y) − ε ≥ r(y,ϕ(t, y)) + (Lϕ(t,y)V ∗m)(t, y)(25)

for s ≤ t ≤ s + δ and y ∈ N . Let

τδ := minτ(x,N, ϕ, s), s + δ(26)

where, for notational simplicity, we omit x, N , ϕ and s.


Hence, for s ≤ t ≤ τδ the process x(t,ϕ) verifies

V ∗m(t, x(t,ϕ)) − ε ≥ r(x(t,ϕ),ϕ(x(t, ϕ))) + (Lϕ(t,x(t,ϕ))V ∗m)(t, x(t, ϕ)).

Multiplying this expression by e−α(t−s), integrating over the interval[s, τδ], and then taking expectation Eϕ,s,x yields

Eϕ,s,x!" τδ

s e−α(t−s)V ∗m(t, x(t,ϕ))dt#− ε

α(1 − Eϕ,s,xe−α(τδ−s))

≥ Eϕ,s,x!" τδ

s e−α(t−s)[r(x(t, ϕ),ϕ(x(t,ϕ)))

+ (Lϕ(t,x(t,ϕ))V ∗m)(t, x(t,ϕ))]dt#.

(27)

Then observe that by the Markov property

V m(ϕ, s, x) = Eϕ,s,x!" τδ

s e−α(t−s)[r(x(t,ϕ), ϕ(x(t,ϕ)))]dt#

+ Eϕ,s,x[e−α(τδ−s)V m(ϕ, τδ, x(τδ, ϕ))],(28)

and also that defining V∗m(t, y) := e−αtV ∗m(t, y) for 0 ≤ t ≤ m and

y ∈ X, by Lemma 6.3 we have

(LaV∗m)(t, y) = e−αt(LaV ∗m)(t, y) − αe−αtV ∗m(t, y).(29)

Hence, applying Dynkin’s formula we obtain

Eϕ,s,x[e−ατδV ∗m(τδ, x(τδ,ϕ))] − e−αsV ∗m(s, x)

= Eϕ,s,x

$" τδ

s (Lϕ(t,x(t,ϕ))V∗m)(t, x(t,ϕ))dt

%.

(30)

Substituting (28), (29) and (30) in (27) gives

V ∗m(s, x) − V m(ϕ, s, x) ≥ εα(1 − Eϕ,s,x[e−α(τδ−s)])

+ Eϕ,s,x[e−α(τδ−s)(V ∗m(τδ, x(τδ, ϕ)) − V m(ϕ, τδ, x(τδ, ϕ)))],

which, by (3), implies that

V ∗m(s, x) − V m(ϕ, s, x) ≥ ε

α(1 − Eϕ,s,x[e−α(τδ−s)]).

By Assumption B(d) there exists some η such that Eϕ,s,x[e−α(τδ−s)] ≤η < 1 for every ϕ ∈ Φ. We deduce that

V ∗m(s, x) − V m(ϕ, s, x) ≥ ε

α(1 − η) for every ϕ ∈ Φ,


which contradicts the definition of V ∗m (see (3)). This establishes state-ment (i).

Proof of (ii). We fix an arbitrary ε > 0. The continuity of the functiona !→ r(x, a) + (LaV ∗m)(s, x) (Assumption B(c)), together with the factthat A(x) is σ-compact and that K is measurable, implies that the hy-potheses of [23, Corollary 4.3] are fulfilled. So, there exists a measurableselector or, in other words, there exists ϕ ∈ Φ such that

αV ∗m(t, y) ≤ r(y, ϕ(t, y)) + (Lϕ(t,y)V ∗m)(t, y) + αε(31)

for 0 ≤ t < m and y ∈ X (recall (24)), and, therefore,

α(V ∗m − ε1)(t, x(t,ϕ)) ≤ r(x(t,ϕ),ϕ(t, x(t, ϕ)))

+ (Lϕ(t,x(t,ϕ))(V ∗m − ε1))(t, x(t,ϕ))(32)

for 0 ≤ t < m, where 1 is the constant function equal to 1. Multiplyingexpression (32) by e−α(t−s), where 0 ≤ s < m, integrating over theinterval t ∈ [s,m] and then taking expectation Eϕ,s,x (for arbitraryx ∈ X) yields, using standard arguments, that

V ∗m(s, x) − ε ≤ V m(ϕ, s, x) + e−α(m−s)(Eϕ,s,x[V ∗m(m,x(m,ϕ))] − ε)

or, equivalently, taking into account (9) (cf. Lemma 6.4),

V ∗m(s, x) ≤ V m(ϕ, s, x) + ε(1 − e−α(m−s))

for 0 ≤ s < m and x ∈ X.(33)

This shows that ϕ ∈ Φ is ε-optimal.

Proof of (iii). To prove (iii) we will also proceed by contradiction. Hencesuppose that there exist x ∈ X and s ∈ [0,m) such that

αV ∗m(s, x) < supa∈A(x)

r(x, a) + (LaV ∗m)(s, x).

Therefore, by Assumption B, there exist β > 0, δ ∈ (0,m − s) and aneighborhood N of x such that

αV ∗m(t, y) + β ≤ supa∈A(y)

r(y, a) + (LaV ∗m)(t, y)

for y ∈ N and s ≤ t ≤ s + δ. Fix arbitrary ε > 0 and choose ϕ ∈ Φ asin (31). Then

αV ∗m(t, y) + β ≤ r(y, ϕ(t, y)) + (Lϕ(t,y)V ∗m)(t, y) + αε(34)


for y ∈ N and s ≤ t ≤ s + δ. Observe that (34) is the same as (25)except for the inequality sign and the value of the constants. Hence,exactly as in the proof of (i) and with τδ defined similarly (see (26)), weobtain

V ∗m(s, x) − V m(ϕ, s, x) ≤ αε−βα (1 − Eϕ,s,x[e−α(τδ−s)])

+ Eϕ,s,x[e−α(τδ−s)(V ∗m(τδ, x(τδ,ϕ)) − V m(ϕ, τδ, x(τδ,ϕ)))].

Recalling (33), this yields that

0 ≤ V ∗m(s, x) − V m(ϕ, s, x)

≤ αε−βα (1 − Eϕ,s,x[e−α(τδ−s)])

+ ε(Eϕ,s,x[e−α(τδ−s)] − e−α(m−s)).

(35)

By Assumption B(d), there exists some η such that Eϕ,s,x[e−α(τδ−s)] ≤η < 1 for all ϕ ∈ Φ and thus, letting ε tend to 0 in (35), we obtain0 ≤ −β

α(1 − η), which is not possible. Therefore, we have proved (iii).

Proof of (iv). Finally, the uniqueness of the solution of (8)–(9) followsfrom standard arguments. More precisely,

αv(s, x) ≥ supa∈A(x)

r(x, a) + (Lav)(s, x) for 0 ≤ s < m and x ∈ X

and (9) imply that V ∗m ≤ v (see Lemma 6.4), whereas the reverseinequality gives (as in the proof of (ii) and recalling that v satisfiesAssumption B(c)) the existence of strategies with payoff arbitrarily closeto v, thus establishing that v = V ∗m. !

Proof of Corollaries 3.2 and 3.3. These results are easily provedusing [23, Corollary 4.3] and Lemma 6.7. !

Proof of Theorem 4.1. Fix a time horizon [0,m], with m > T , and anarbitrary strategy ϕ ∈ Φ. By Theorem 3.1 we have that, for 0 ≤ t < mand y ∈ X,

αV ∗m(t, y) ≥ r(y, ϕ(t, y)) + (Lϕ(t,y)V ∗m)(t, y).(36)

Recalling Lemma 6.6 we also have

αV (ϕ, t, y) = r(y, ϕ(t, y)) + (Lϕ(t,y)V (ϕ, ·, ·))(t, y)(37)


for 0 ≤ t < m and y ∈ X. Substracting (37) from (36) gives that for0 ≤ t < m and y ∈ X

αV ∗m(t, y) − αV (ϕ, t, y) ≥ (Lϕ(t,y)(V ∗m − V (ϕ, ·, ·)))(t, y)

or, equivalently, (Lϕ(t,y)V )(t, y) ≤ 0, where we define

V (t, y) := e−αt(V ∗m(t, y) − V (ϕ, t, y))

(recall Lemma 6.3). Using Dynkin’s formula we derive that

Eϕ,s,x[e−αm(V ∗m(m,x(m,ϕ)) − V (ϕ,m, x(m,ϕ)))]

≤ e−αs(V ∗m(s, x) − V (ϕ, s, x)).

By Theorem 3.1, we have that V ∗m(m,x(m, ϕ)) = 0 and thus, for everyx ∈ X, 0 ≤ s ≤ T and ϕ ∈ Φ,

−Eϕ,s,x[e−αmV (ϕ,m, x(m,ϕ))] ≤ e−αs(V ∗m(s, x) − V (ϕ, s, x)).

Using properties of the conditional expectation, one can easily showthat

Eϕ,s,x[e−αmV (ϕ,m, x(m,ϕ))]

= Eϕ,s,x!" ∞

m e−αtr(x(t,ϕ),ϕ(t, x(t,ϕ)))dt#.

Now, by Lemma 6.2, for x ∈ X and s ∈ [0, T ],

|Eϕ,s,x[e−αmV (ϕ,m, x(m,ϕ))]| ≤ W (x)H(T, m),

where

H(T,m) := M

$ecT

%e−(α−c)m

α − c(1 + (b/c))

&+ (b/c)

e−αm

α

',(38)

with b, c and M as in Assumption A. Therefore, for every ϕ ∈ Φ, x ∈ Xand s ∈ [0, T ],

V (ϕ, s, x) − V ∗m(s, x) ≤ W (x)H(T, m)eαT .(39)

Taking the supremum in (39) over ϕ ∈ Φ and recalling Lemma 6.1 gives

V ∗(x) − V ∗m(s, x) ≤ W (x)H(T, m)eαT(40)

for x ∈ X and 0 ≤ s ≤ T .


To prove the theorem we will show that an inequality similar to (40)holds, but now with a lower bound.

Fix a positive number ε and let ϕ be ε-optimal for the finite horizonproblem over the time horizon [0, m], that is,

V m(ϕ, s, x) + ε ≥ V ∗m(s, x) for every x ∈ X and s ∈ [0,m].

On the other hand,

V (ϕ, s, x)

= V m(ϕ, s, x) + Eϕ,s,x

!" ∞

me−α(t−s)r(x(t,ϕ),ϕ(t, x(t,ϕ)))dt

#

≤ V ∗(x)

and, therefore, using the bound (39), we obtain

V ∗m(s, x) − H(T, m)W (x)eαT ≤ V ∗(x) + ε

for every x ∈ X, ε > 0 and 0 ≤ s ≤ T . Since ε is arbitrary, this yields,together with (40),

sup0≤s≤T

||V ∗m(s, ·) − V ∗||W ≤ H(T,m)eαT for m > 0.

Since H(T, m) converges to zero when T is fixed and m → ∞, we obtainthe stated result. !

Proof of Corollary 4.3. By Assumption B(a), x '→ V ∗m(0, x) is acontinuous function. Since W is bounded on compact sets, it followsthat V ∗m(0, ·)m>0 converges uniformly to V ∗ on compact sets. Recall-ing that X is locally compact, this proves that V ∗ is continuous. !

Remark 7.1 Let us make some final comments on these results. Ifwe do not impose Assumption C then, in the proof of Theorem 4.1,expressions (36) and (37) would be valid only for stationary strategiesf ∈ F. Thus defining V ∗(x) := supf∈ V (f, x) and recalling Lemma 6.5,inequality (40) would become

V ∗(x) − V ∗m(s, x) ≤ H(T, m)W (x)eαT for x ∈ X and 0 ≤ s ≤ T .

The rest of the proof of Theorem 4.1 remains valid, and then

−H(T,m)W (x)eαT ≤ V ∗(x) − V ∗m(s, x) for x ∈ X and 0 ≤ s ≤ T .


It follows that if we knew in advance that V ∗ = V ∗, then Theorem 4.1would be verified without imposing Assumption C. However, the (wellknown) fact that V ∗ = V ∗ is derived using the infinite horizon optimalityequation, which has not yet been proved.

Proof of Theorem 4.4. First of all let us prove that for every x ∈ X

limm→∞

supf∈

|(Lf(x)V ∗m)(0, x) − (Lf(x)V ∗)(x)| = 0.(41)

We will proceed by contradiction. Hence, suppose that there exist x ∈X, ε > 0 and fm ∈ F such that

|(Lfm(x)V ∗m)(0, x) − (Lfm(x)V ∗)(x)| > ε

for infinitely many values of m. Therefore, for infinitely many m, either

(Lfm(x)V ∗m)(0, x) − (Lfm(x)V ∗)(x) > ε(42)

or(Lfm(x)V ∗m)(0, x) − (Lfm(x)V ∗)(x) < −ε.(43)

Suppose for instance that (42) holds. By Assumption D(b) there existsδ > 0 and a neighborhood N of x such that

(Lfm(y)V ∗m)(t, y) − (Lfm(y)V ∗)(y) ≥ ε/2(44)

for 0 ≤ t ≤ δ and y ∈ N , for infinitely many m. Observe that, byequicontinuity, neither δ nor N depend on m. Let τm be defined byτm := minτ(x,N, fm), δ and observe that (44) implies that for 0 ≤t ≤ τm

(Lfm(x(t,fm))V ∗m)(t, x(t, fm)) − (Lfm(x(t,fm))V ∗)(x(t, fm)) ≥ ε/2.

As a consequence of Dynkin’s formula we obtain that

Efm,x[V ∗m(τm, x(τm, fm)) − V ∗(x(τm, fm))] − [V ∗m(0, x) − V ∗(x)]

≥ ε2Efm,x[τm].

It follows from Theorem 4.1, the fact that τm ≤ δ and Lemma 6.2, thatthe left-hand side of the above inequality is less than or equal to

!ecδW (x) +

b

c(ecδ + 1)

"H(δ,m)eαδ + W (x)H(0,m).


On the other hand, Assumption B(d) ensures that there exists someη > 0 such that Efm,x[τm] ≥ η for all m. Therefore, for infinitelymany m

!ecδW (x) +

b

c(e|c|δ + 1)

"H(δ,m)eαδ + W (x)H(0,m) ≥ εη

2,

which is not possible because H(δ,m) and H(0,m) tend to zero as m→∞.A similar conclusion is obtained if (43) holds for infinitely many m.

Therefore, (41) shows that for fixed x ∈ X

r(x, a) + (LaV ∗m)(0, x) → r(x, a) + (LaV ∗)(x)

uniformly in a ∈ A(x) as m → ∞, and thus

limm→∞

supa∈A(x)

r(x, a) + (LaV ∗m)(0, x) = supa∈A(x)

r(x, a) + (LaV ∗)(x).

This also yields that

a %→ r(x, a) + (LaV ∗)(x) is continuous on A(x).(45)

Hence, letting m → ∞ in the finite horizon optimality equation

αV ∗m(0, x) = supa∈A(x)

r(x, a) + (LaV ∗m)(0, x)

shows that

αV ∗(x) = supa∈A(x)

r(x, a) + (LaV ∗)(x) for x ∈ X.

The existence of ε-optimal strategies is derived, as in the proof ofTheorem 3.1, using (45) and [23, Corollary 4.3]. The uniqueness of thesolution is proved as in Theorem 3.1.

This completes the proof of Theorem 4.4. !

Proof of Theorem 4.5. The proof mimics that of Theorem 3.1 ex-cept for some technical details derived from the fact that the optimalityequation does not depend on the time component. However, for com-pleteness of the exposition, we give a full proof of the theorem.

We will show that V ∗ satisfies (10) and then we will establish theuniqueness property. First of all, let us prove that

αV ∗(x) ≤ supa∈A(x)

r(x, a) + (LaV ∗)(x) for every x ∈ X.(46)


We will proceed by contradiction. Hence, suppose that there exists somex ∈ X such that

αV ∗(x) > supa∈A(x)

r(x, a) + (LaV ∗)(x).

By Assumption E(b), there exists some ε > 0 and a neighborhood N ofx with

αV ∗(y) − ε ≥ supa∈A(y)

r(y, a) + (LaV ∗)(y) for every y ∈ N .(47)

Given arbitrary ϕ ∈ Φ and T > 0, define

τ(ϕ, T ) := minτ(x, N, ϕ), T(48)

where for notational convenience, we have omitted x and N .By (47), for every 0 ≤ t ≤ τ(ϕ, T ) we have

αV ∗(x(t,ϕ)) − ε ≥ r(x(t, ϕ),ϕ(x(t,ϕ))) + (Lϕ(t,x(t,ϕ))V ∗)(x(t,ϕ)).

Multiplying both sides of the above inequality by e−αt, integrating overthe interval [0, τ(ϕ, T )], and then taking expectation Eϕ,x yields

Eϕ,x

!" τ(ϕ,T )0 αe−αtV ∗(x(t,ϕ))dt

#− ε

α(1 − Eϕ,x[e−ατ(ϕ,T )])

≥ Eϕ,x

!" τ(ϕ,T )0 e−αt[r(x(t,ϕ),ϕ(x(t,ϕ)))

+ (Lϕ(t,x(t,ϕ))V ∗)(x(t,ϕ))]dt ] .

(49)

Now observe that the Markov property gives

V (ϕ, x) = Eϕ,x

!" τ(ϕ,T )0 e−αtr(x(t,ϕ),ϕ(x(t,ϕ)))dt

#

+ Eϕ,x[e−ατ(ϕ,T )V (ϕ, x(τ(ϕ, T ),ϕ))].(50)

Observe also that the extended generator applied to the function V∗

defined as

V∗(t, y) := e−αtV ∗(y) for t ≥ 0 and y ∈ X,

verifies (by Lemma 6.3) that for t ≥ 0 and y ∈ X

(Lϕ(t,y)V∗)(t, y) = e−αt(Lϕ(t,y)V ∗)(y) − αe−αtV ∗(y)


and also that, by Dynkin’s formula,

Eϕ,x[e−ατ(ϕ,T )V ∗(x(τ(ϕ, T ),ϕ))] − V ∗(x)

= Eϕ,x

!" τ(ϕ,T )0 (Lϕ(t,x(t,ϕ))V

∗)(t, x(t,ϕ))dt#.

(51)

Substituting (50) and (51) in the inequality (49) shows that

V ∗(x) − V (ϕ, x) − ε

α(1 − Eϕ,x[e−ατ(ϕ,T )]) ≥

Eϕ,x[e−ατ(ϕ,T )[V ∗(x(τ(ϕ, T ),ϕ)) − V (ϕ, x(τ(ϕ, T ),ϕ))]].(52)

Since V ∗(·) ≥ V (ϕ, ·), (52) gives

V ∗(x) − V (ϕ, x) ≥ ε

α(1 − Eϕ,x[e−ατ(ϕ,T )]).

If T ↑ ∞ then τ(ϕ, T ) ↑ τ(x,N,ϕ) and, by monotone convergence,

V ∗(x) − V (ϕ, x) ≥ ε

α(1 − Eϕ,x[e−ατ(x,N,ϕ)]).

Therefore, by Assumption E(d), there exists some δ > 0 such thatV ∗(x) − V (ϕ, x) ≥ δ for every ϕ ∈ Φ, which contradicts the definitionof V ∗. This establishes (46).

Before proceeding to prove the reverse inequality, that is,

αV ∗(x) ≥ supa∈A(x)

r(x, a) + (LaV ∗)(x) for every x ∈ X,(53)

we need first to prove the existence of ε-optimal strategies.Given ε > 0, [23, Corollary 4.3] and Assumption E(c) ensure that

there exists some f ∈ F for which

αV ∗(x) ≤ r(x, f(x)) + (Lf(x)V ∗)(x) + εα for every x ∈ X,

which implies (Lemma 6.5) that f is ε-optimal.Now we are ready to prove (53). Again, we proceed by contradiction

supposing that there exists some x ∈ X such that

αV ∗(x) < supa∈A(x)

r(x, a) + (LaV ∗)(x).

By Assumption E(b), for some β > 0 and a neighborhood N of x,

αV ∗(y) + β ≤ supa∈A(y)

r(y, a) + (LaV ∗)(y) for every y ∈ N.


Given arbitrary ε > 0, let f ∈ F be such that

αV ∗(y) + β ≤ r(y, f(y)) + (Lf(y)V ∗)(y) + αε for every y ∈ N.

Exactly as before (only the inequality sign has changed) we can showthat (cf. (52))

V ∗(x) − V (f, x) + β−εα (1 − Ef,x[e−ατ(f,T )])

≤ Ef,x[e−ατ(f,T )[V ∗(x(τ(f, T ), f)) − V (f, x(τ(f, T ), f))]],(54)

where τ(f, T ) is defined as in (48). Now, by ε-optimality, V ∗(·) ≤V (f, ·) + ε and so, (54) implies

V ∗(x) − V (f, x) +β − ε

α(1 − Ef,x[e−ατ(f,T )]) ≤ εEf,x[e−ατ(f,T )].

Letting T → ∞ and rearranging terms, we obtain

V ∗(x) − V (f, x)

≤ −βα(1 − Ef,x[e−ατ(x,N,f))]) + ε

!1α + 1+α

α Ef,x[e−ατ(x,N,f)]".

Taking the lim supε→0 in this expression, recalling that f is ε-optimaland that (by Assumption E(d)) Ef,x[e−ατ(x,N,f)] is bounded away from 1(uniformly in f), yields a contradiction.

This establishes (53) which, together with (46), proves that V ∗ sat-isfies (10).

The uniqueness property is proved as in Theorem 4.4. !

Remark 7.2 Observe that Theorem 4.5 is verified if we replace thecontinuity condition on V ∗ (Assumption E(b)) with the following con-ditions:

(i) V ∗ is lower semicontinuous,

(ii) for every f ∈ F, V (f, ·) is continuous.

The reason for this is that proving (46) only requires lower semiconti-nuity of V ∗. Then the existence of ε-optimal stationary strategies showsthat V (fn, ·) converges uniformly to V ∗ for some sequence fn ⊆ F,thus establishing that V ∗ is continuous, which is needed to prove (53).

Proof of Corollaries 4.6 and 4.7. These results are easily provedusing [23, Corollary 4.3] and Lemma 6.7. !


8 Proof of results in Section 5

Finite horizon controlled diffusions.

Lemma 8.1 If Assumption CD holds, then Assumption B(a) is satis-fied.

Proof. First of all, observe that V ∗T as defined in (13) is not the supre-mum of the expected reward of the family of Markov strategies, asdefined in (3). However, as a consequence of [18, Theorem 5.1.2] (inparticular, Assumption CD(d) is used)

V ∗T (s, x) = supϕ∈Φ

V T (ϕ, s, x) for every 0 ≤ s ≤ T and x ∈ R.

It is worth noting that the same result can be obtained replacing As-sumption CD(d) with other nondegeneracy conditions; see [18, p. 214].

By [18, Theorem 4.7.7], we have that V ∗T is continuous on [0, T ]×Rand also that V ∗T has bounded first derivative with respect to t ∈ [0, T ]and bounded second derivative with respect to x ∈ R . Thus, V ∗T ∈∩ϕ∈ΦDϕ follows. !

Lemma 8.2 If Assumption CD is satisfied and, moreover, the functionV ∗T is in C1,2([0, T ] × R), then Assumptions B(b) and B(c) hold.

Proof. The result is easily proved using Assumption CD and Lemma 6.7.!

Lemma 8.3 Assumption CD implies Assumption B(d).

Proof. Fix x ∈ R, a neighborhood N of x and s ≥ 0. To simplify thenotation, and without loss of generality, we will assume that s = 0.Suppose that (x− ε, x+ ε) ⊆ N for some ε > 0. Let us prove that thereexists δ > 0 such that

infϕ∈Φ

Pϕ,xτ(x, N, ϕ) > δ > 0.

Recall that

x(t,ϕ) = x +! t

0b(s, x(s,ϕ),ϕ(s, x(s, ϕ)))ds

+! t

0σ(s, x(s,ϕ),ϕ(s, x(s,ϕ)))dWs


for 0 ≤ t ≤ T . Observe now that

τ(x,N,ϕ) ≤ δ ⊆ sup0≤t≤δ

|x(t,ϕ) − x| ≥ ε,

and that sup0≤t≤δ

|x(t,ϕ) − x| ≥ ε ⊆ D1(δ) ∪ D2(δ),

where

D1(δ) :=!" δ

0|b(s, x(s,ϕ),ϕ(s, x(s,ϕ)))|ds ≥ ε/2

#

and

D2(δ) :=

$sup

0≤t≤δ

%%%%" t

0σ(s, x(s,ϕ),ϕ(s, x(s,ϕ)))dWs

%%%% ≥ ε/2

&.

Suppose that 0 < δ ≤ ε/4K (where K is as in Assumption CD(b)), andso

D1(δ) ⊆!" δ

0|x(s,ϕ)|ds ≥ ε/4K

#⊆

!" δ

0x2(s,ϕ)ds ≥ ε2/16δK2

#,

where the first inclusion follows from Assumption CD(b) and the secondone from Jensen’s inequality. Therefore, from Chebychev’s inequality,

Pϕ,x(D1(δ)) ≤16δK2

ε2Eϕ,x

'" δ

0x2(s, ϕ)ds

(.

On the other hand, by the martingale property of the stochastic integralsand Ito’s isometry (see [21, Corollary 3.2.6])

Pϕ,x(D2(δ)) ≤ 4K2

ε2Eϕ,x

'" δ

0(1 + |x(s,ϕ)|)2ds

(

≤ 8δK2

ε2+

8K2

ε2Eϕ,x

'" δ

0x2(s,ϕ)ds

(.

Define now

G(ϕ, x, δ) := Eϕ,x

'" δ

0x2(s,ϕ)ds

(.

So far we have shown that

Pϕ,xτ(x,N,ϕ) ≤ δ ≤ 8δK2

ε2+

16δK2 + 8K2

ε2G(ϕ, x, δ).(55)


Therefore, to prove the stated result, it suffices to show that

limδ→0

supϕ∈Φ

G(ϕ, x, δ) = 0.(56)

It follows from standard arguments (see the proof of Theorem 5.2.1 in[21]) that

Eϕ,x[x2(s,ϕ)] ≤ 4K2(T + 1)T + 4K2(T + 1)! s

0Eϕ,x[x2(u,ϕ)]du

and, from Gronwall’s inequality [21, p. 78],

Eϕ,x[x2(s,ϕ)] ≤ 4K2(T + 1)Te4K2(T+1)T =: ∆ for 0 ≤ s ≤ T ,

which proves that G(ϕ, x, δ) ≤ ∆δ, and (56) follows. Lemma 6.8 to-gether with (55) completes the proof. !

Proof of Theorem 5.1. The proof follows from Lemmas 8.1–8.3, The-orem 3.1 and Corollary 3.2. !

Infinite horizon controlled Markov chains.

Proof of Theorem 5.2.Proof of (i). Assumptions E(a) and E(b) are easily proved (recall thatthe state space X is endowed with the discrete topology). Note howeverthat the continuity of a "→ r(x, a)+(LaV ∗)(x) stated in Assumption E(c)is not necessary because this property was used to derive the existenceof a measurable selector. To prove that Assumption E(d) holds, we shallprove that Lemma 6.8, which implies Assumption B(d), which in turnimplies Assumption E(d), is satisfied.

Let x ∈ X, s ≥ 0 and N = x, which is a neighborhood of x. Then

Pϕ,s,xτ(x,N, ϕ, s) ≥ s + δ0 = exp! s+δ0

sqxx(u,ϕ)du

≥ exp−q(x)δ0,

where q(x) is defined as in Assumption CMC(a). Thus

infϕ∈Φ

Pϕ,s,xτ(x,N,ϕ, s) ≥ s + δ0 > 0

and so, Lemma 6.8 is verified. This shows that Assumptions CMC(a)–(b) imply Assumption E.


Proof of (ii). It suffices to show that Assumption E(c) is satisfied, thatis, we must prove that for each x ∈ X, the function

a "→!

y∈X

q(y|x, a)V ∗(y)(57)

is continuous on A(x). For each integer p ≥ 0 we know from Assump-tion CMC(c) that a "→

"0≤y≤p q(y|x, a)V ∗(y) is continuous on A(x).

Therefore, if we prove that

limp→∞

supa∈A(x)

#####!

y>p

q(y|x, a)V ∗(y)

##### = 0,

then the continuity of (57) will follow. To this end, we will show that ifp ≥ x then

limp→∞

supa∈A(x)

!

y>p

q(y|x, a)W (y) = 0.(58)

Observe that the decreasing sequence of continuous functions (indexedby p ≥ x)

a "→!

y>p

q(y|x, a)V ∗(y)

converges to zero and, therefore, by Dini’s theorem, convergence is uni-form on the compact set A(x), thus proving (58). This completes theproof. !

Tomas Prieto-RumeauDepartmento de Estadıstica,Facultad de Ciencias, UNED,c/ Senda del Rey 9,28040 Madrid, Spain,[email protected]

Onesimo Hernandez-LermaDepartamento de Matematicas,CINVESTAV-IPN,Apartado Postal 14-740,Mexico D.F. 07000, Mexico,[email protected]

References

[1] Almudevar A., A dynamic programming algorithm for the optimalcontrol of Markov piecewise deterministic processes, SIAM J. Con-trol Optim. 40 (2002), 525–539.

[2] Bensoussan A.; Robin M., On the convergence of the discrete timedynamic programming equation for general semigroups, SIAM J.Control Optim. 20 (1982), 722–746.


[3] Berge C., Topological Spaces, Macmillan, New York, 1963.

[4] Bhatt A. G.; Borkar V. S., Occupation measures for controlledMarkov processes: characterization and optimality, Ann. Probab.24 (1996), 1531–1562.

[5] Dempster M. A. H.; Ye J. J., Necessary and sufficient optimalityconditions for control of piecewise deterministic Markov processes,Stoch. Stoch. Rep. 40 (1992), 125–145.

[6] Doshi B. T., Continuous-time control of Markov processes on anarbitrary state space: discounted rewards, Ann. Statist. 4 (1976),1219–1235.

[7] Doshi B. T., Continuous-time control of Markov processes on anarbitrary state space: average return criterion, Stoch. Proc. Appl.4 (1976), 55–77.

[8] Down D.; Meyn S. P.; Tweedie R. L., Exponential and uniform er-godicity of Markov processes, Ann. Probab. 23 (1995), 1671–1691.

[9] Feller W., On the integro-differential equations of purely discon-tinuous Markoff processes, Trans. Amer. Math. Soc. 48 (1940),488–515.

[10] Fleming W. H.; Rishel R. W., Deterministic and Stochastic Opti-mal Control, Springer, New York, 1975.

[11] Fleming W. H.; Soner H. M., Controlled Markov Processes andViscosity Solutions, Springer, New York, second edition, 2006.

[12] Guo X. P., Continuous-time Markov decision processes with dis-counted rewards: the case of Polish spaces, Math. Oper. Res.(2007), 73–87.

[13] Guo X. P.; Hernandez-Lerma O., Continuous-time controlled Mar-kov chains with discounted rewards, Acta Appl. Math. 79 (2003),195–216.

[14] Hernandez-Lerma O., Lectures on Continuous-Time Markov Con-trol Processes, Sociedad Matematica Mexicana, Mexico City, 1994.

[15] Hernandez-Lerma O.; Govindan T. E., Nonstationary continuous-time Markov control processes with discounted costs on infinitehorizons, Acta Appl. Math. 67 (2001), 277–293.


[16] Hernandez-Lerma O.; Lasserre J. B., Discrete-Time Markov Con-trol Processes: Basic Optimality Criteria, Springer, New York,1996.

[17] Hernandez-Lerma O.; Lasserre J. B., Further Topics on Discrete-Time Markov Control Processes, Springer, New York, 1999.

[18] Krylov N. V., Controlled Diffusion Processes, Springer, New York,1980.

[19] Kurtz T. G.; Stockbridge R. H., Existence of Markov controls andcharacterization of optimal Markov controls, SIAM J. Control Op-tim. 36 (1998), 609–653. Erratum, ibid. 37 (1998), 1310–1311.

[20] Meyn S. P.; Tweedie R. L., Stability of Markovian processes III:Foster-Lyapunov criteria for continuous-time processes, Adv. Appl.Prob. 25 (1993), 518–548.

[21] Øksendal B., Stochastic Differential Equations, Fifth Edition,Springer, New York, 1998.

[22] Pliska S. R., Controlled jump processes, Stoch. Proc. Appl. 3(1975), 259–282.

[23] Rieder U., Measurable selection theorems for optimization prob-lems, Manuscripta Math. 24 (1978), 115–131.

[24] Stockbridge R. H., Time-average control of martingale problems:existence of a stationary solution, Ann. Probab. 18 (1990), 190–205.

[25] Stockbridge R. H., Time-average control of martingale problems: alinear programming formulation, Ann. Probab. 18 (1990), 206–217.


On bounds for the stability number of graphs ∗

Isidoro Gitler Carlos E. Valencia

Abstract

Let G be a graph without isolated vertices and let α(G) be itsstability number and τ(G) its covering number. In this paperwe study the minimum number of edges a connected graph canhave as a function of α(G) and τ(G). In particular we obtain thefollowing lower bound:

q(G) ≥ α(G) − c(G) + Γ(α(G), τ(G)),

where c(G) is the number of connected components of G and

Γ(a, t) = min

a

i=1

zi

2

z1 + · · · + za = a + t

and zi ≥ 0 ∀ i = 1, . . . , a

,

for a and t two arbitrary natural numbers.Also we prove that α(G) ≤ τ(G)[1 + δ(G)], where δ(G) =

α(G) − σv(G) and σv(G) is the σv-cover number of a graph, thatis, the maximum natural number m such that every vertex of Gbelongs to a maximal independent set with at least m vertices.

2000 Mathematics Subject Classification: 05C69, 05C35.Keywords and phrases: q-minimal graph, stability number, coveringnumber.

1 Preliminaries

Let G = (V, E) be a graph with |V | = n vertices and |E| = q edges. IfU ⊆ V is a subset of vertices, then the induced subgraph on U , denoted

∗Invited Article. Authors partially supported by CONACyT grant 49835 andSNI.

41


42 I. Gitler and C. Valencia

by G[U ], is the graph with U as a vertex set and whose edges areprecisely the edges of G with both ends in U .

A subset A ⊂ V is a minimal vertex cover for G if: (i) every edge ofG is incident with at least one vertex in A, and (ii) there is no propersubset of A with the first property. If A satisfies condition (i) only, thenA is called a vertex cover of G.

The vertex covering number of G, denoted by τ(G), is the number ofvertices in a minimum vertex cover in G, that is, the size of any smallestvertex cover in G.

It is convenient to regard the empty set as a minimal vertex coverfor a graph with all its vertices isolated.

A subset M of V is called a stable set if no two vertices in M areadjacent. We call M a maximal stable set if it is maximal with respectto inclusion. The stability number of a graph G is given by

α(G) = max|M | |M ⊂ V (G) is a stable set in G.

Note that a set of vertices in G is maximal stable set if and only ifits complement is a minimal vertex cover for G.

Thus we have α(G) + τ(G) = n.A subset W of V is called a clique if any two vertices in W are

adjacent. We call W maximal if it is maximal with respect to inclusion.The clique number of a graph G is given by

ω(G) = max|W | |W ⊂ V (G) is a clique in G.

Given a subset U ⊂ V , the neighbour set of U , denoted by N(U), isdefined as N(U) = v ∈ V | v is adjacent to some vertex in U.

2 The number of edges of a connected graphwith fixed stability number

We give in Theorem 2.3 a lower bound for the number of edges of a graphG as a function of the stability number α(G), the covering number τ(G)and the number of connected components c(G) of G. This is an answerto an open question posed by Ore in his book [6] which is a variant forconnected graphs of a celebrated theorem of Turan [7].

For a graph G = (V, E), we will denote by q(G) the cardinality ofthe edge set E(G) of G. We say that a connected graph G is q-minimalif there is no graph G′ such that

On bounds for the stability number of graphs 43

(i) G′ is connected,(ii) α(G′) = α(G),(iii) τ(G′) = τ(G) and(iv) q(G′) < q(G).

Hence if G is q-minimal, then either α(G) < α(G − e) or c(G) <c(G− e) for all the edges e of G (note that α(G) < α(G− e) if and onlyif τ(G) > τ(G − e)). That is, an edge of a q-minimal graph is eitherα-critical or a bridge. Therefore the blocks of a q-minimal graph areα-critical graphs. Here an edge e of a graph G is α-critical if α(G−e) =α(G) + 1, G is α-critical if all the edges of G are α-critical and is aτ -critical graph if τ(G − v) = τ(G) − 1 for all the vertices v of G.

In order to bound the number of edges we introduce the followingnumerical function. Let a and t be two natural numbers and let

Γ(a, t) = min

!a"

i=1

#zi

2

$ %%%%%z1 + · · · + za = a + t

and zi ≥ 0 ∀ i = 1, . . . , a

&,

Lemma 2.1 Let a and t be natural numbers, then

(i) Γ(a, t) = (a − s)#

r2

$+ s

#r + 1

2

$where a + t = r(a) + s with

0 ≤ s < a.

(ii) Γ(a−1, t)−Γ(a, t) ≥ 12(⌊a+t

a ⌋2−⌊a+ta ⌋) ≥ 0 for all a ≥ 2 and t ≥ 1.

Moreover we have that Γ(a− 1, t)−Γ(a, t) = 12(

'a+ta

(2 −'

a+ta

() if

and only if'

a+ta

(≥ t

a−1 and we have that'

a+ta

(2 −'

a+ta

(= 0 if

and only if 0 ≤ t < a.

(iii) Γ(a, t) − Γ(a, t − 1) = 1 +'

t−1a

(=

)ta

*for all a ≥ 1 and t ≥ 2.

(iv)+k

i=1 Γ(ai, ti) ≥ Γ(+k

i=1 ai,+k

i=1 ti) for all ai ≥ 1 and ti ≥ 1.

Furthermore we have that

Γ(a1, t1) + Γ(a2, t2) = Γ(a1 + a2, t1 + t2)

if and only if,

t1a1

-=

,t2a2

-.

(v).

2(a−1+Γ(a,t))a+t

/= 1 +

'ta

(+ L, where L = −1 if and only if a = 1,

L = 0 if and only if either a | t (a does not divide t) or a|t (a dividest) and a ≥ 2 and L = 1 if either 1 ≤ t < a or a + 2 ≤ t < 2a.


Proof: (i) The case for a = 1 is trivial. For a ≥ 2 we will use the nextresult.

Claim 2.2 Let n,m ≥ 1 be natural numbers with n > m + 1, then!

n2

"+

!m2

">

!n − 1

2

"+

!m + 1

2

".

Proof: It follows easily, since!

n2

"−

!n − 1

2

"= n − 1.

Let a ≥ 2 and t ≥ 1 be fixed natural numbers, (z1, . . . , za) ∈ Na

such that#a

i=1 zi = a + t and let L(z1, . . . , za) =#a

i=1

!zi

2

". Now, if

z1, . . . , za = r, . . . , r$ %& 'a−s

, r + 1, . . . , r + 1$ %& 's

where a + t = r(a) + s with 0 ≤ s < a, then there exist zi1 and zi2 withzi1 > zi2 + 1. Applying Claim 2.2 we have that

L(z1, . . . za) > L(z1, . . . , zi1 − 1, . . . , zi2 + 1, . . . , za) ≥ Γ(a, t),

and therefore we obtain the result.

(ii) Let a + t = ar + s with r ≥ 1 and 0 ≤ s < a, then

a + t − 1 = (a − 1)(r + l) + s′

where r + s − 1 = (a − 1)l + s′ with l ≥ 0 and 0 ≤ s′ < a − 1.Using part (i) and after some algebraic manipulations we obtain that

2(Γ(a − 1, t) − Γ(a, t)) = (r2 − r) + (l2 − l)(a − 1) + 2ls′.

Therefore Γ(a− 1, t)− Γ(a, t) ≥ 12(⌊a+t

a ⌋2 − ⌊a+ta ⌋) ≥ 0, since r, l, s′ ≥ 0

and u2−u ≥ 0 for all u ≥ 0. Moreover we have that Γ(a−1, t)−Γ(a, t) =12(⌊a+t

a ⌋2 − ⌊a+ta ⌋) if and only if

(l, s′) =(

(0, s′)(1, 0)

These two possibilities imply that r + s < a and r + s = a, re-spectively. Finally it is clear that ⌊a+t

a ⌋2 − ⌊a+ta ⌋ = 0 if and only if

0 ≤ t < a.


(iii) Let a + t − 1 = ar + s with r ≥ 1 and 0 ≤ s < a, then

a + t =

ar + (s + 1) if 0 ≤ s < a − 1a(r + 1) if s = a − 1

and by (i) we have that

Γ(a, t) − Γ(a, t − 1) =

=

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(a − s − 1)(

r2

)+ (s + 1)

(r + 1

2

)− (a − s)

(r2

)+ s

(r + 1

2

)

a

(r + 1

2

)−

(r2

)+ (a − 1)

(r + 1

2

)

=(

r + 12

)−

(r2

)= r =

⌊a + t − 1

a

⌋.

(iv) Follows directly from the definition of Γ(a, t).

(v) Let a+ t = ar + s with r ≥ 1 and 0 ≤ s < a then, by (i) we havethat

⌈2(a − 1 + Γ(a, t))

a + t

⌉=

⎡

⎢⎢⎢⎢⎢

2(

a − 1 + (a − s)(

r2

)+ s

(r + 1

2

))

a + t

⎤

⎥⎥⎥⎥⎥

=⌈

2(a − 1) + (a − s)r(r − 1) + s(r + 1)ra + t

⌉

=⌈

2(a − 1) + r(ar + s) − r(a − s)ar + s

⌉

= r +⌈

2(a − 1) − r(a − s)ar + s

⌉

=⌊

a + t

a

⌋+

⌈2(a − 1) − r(a − s)

ar + s

⌉


Finally

L =⌈

2(a − 1) − r(a − s)ar + s

⌉

=

⎧⎪⎨

⎪⎩

−1 if a = 1,

0 if a | t or a|t, and a ≥ 2,

1 if either r = 1 and s ≥ 1 or r = 2 and s ≥ 2.

Since

L ≥ −1 ⇔ −2(ar + s) < r(s − a) + 2(a − 1)⇔ 2 < (a + s)(2 + r)⇔ a, r ≥ 1,

L ≥ 0 ⇔ −(ar + s) < r(s − a) + 2(a − 1)⇔ 2 < s(r + 1) + 2a⇔ s > 0 or s = 0 and a ≥ 2,

L ≥ 1 ⇔ 0 < r(s − a) + 2(a − 1)⇔ 0 < (a − 1)(2 − r) + r(s − 1)⇔ r = 1, s ≥ 1 or r = 2, s ≥ 2.

Theorem 2.3 ([4, Theorem 3.3]) Let G be a graph, then

q(G) ≥ α(G) − c(G) + Γ(α(G), τ(G)).

Proof: We will use induction on τ(G), the covering number of G.For τ(G) = 1 it is easy to see that the unique connected graphs with

τ(G) = 1 are the stars K1,n (α(K1,n) = n − 1) and the result follows,since

q(K1,n) = n − 1 = (n − 1) − 1 + 1 = α(K1,n) + c(K1,n) + Γ(n − 1, 1).

In the same way it is easy to see that the unique graphs G with α(G) = 1are the complete graphs Kn (τ(Kn) = n − 1). Since we have that,

q(Kn) =(

n2

)= 1 − 1 +

(n2

)= α(Kn) + c(Kn) + Γ(1, n − 1),

it follows that the family of complete graphs satisfies the result.


Moreover the graphs of both families are q-minimal graphs. So wecan assume that the result is true for τ(G) ≤ k > 1.

Let G be a q-minimal graph with τ(G) = k + 1. Since q(G) =!si=1 q(Gi), α(G) =

!si=1 α(Gi) and τ(G) =

!si=1 τ(Gi) where G1, . . . ,

Gs are the connected components of G, it follows from Lemma 2.1(iv)that we can assume with out loss of generality that G is connected andα(G) ≥ 2.

Let e be an edge of G and consider the graph G′ = G − e. We havetwo possibilities

τ(G′) ="

τ(G)τ(G) − 1

That is, an edge of G is either a bridge or α-critical.

Case 1 First assume that G has no bridges, that is, G is a α-criticalgraph. Let v be a vertex of G of maximum degree. Since any α-criticalgraph is τ -critical we have that τ(G−v) = τ(G)−1 and α(G−v) = α(G),moreover since the α-critical graphs are blocks we have that G − v isconnected. Now, by the induction hypothesis we have that

q(G − v) ≥ α(G) − 1 + Γ(α(G), τ(G) − 1).

Using the formula

α(G−v)+τ(G−v)#

i=1

deg(vi) = 2q(G − v)

we conclude that there must exist a vertex v′ ∈ V (G − v) with

deg(v′) ≥$

2q(G − v)α(G − v) + τ(G − v)

%

≥$

2(α(G) − 1 + Γ(α(G), τ(G) − 1))α(G) + τ(G) − 1

%.

Now by Lemma 2.1(iii) and (v) we have that

q(G) = q(G − v) + deg(v)(1)≥ α(G) − 1 + Γ(α(G), τ(G) − 1) + deg(v′)≥ α(G) − 1 + Γ(α(G), τ(G)).

So, if the graph G has an edge that is a bridge, we have that c(G′) =c(G)−1 = 2. Denote by G1 and G2 the connected components of G−e.We have two more cases:


Case 2 Assume that τ(G1) > 0 or τ(G2) > 0, then τ(G1) ≤ k andτ(G2) ≤ k and by the induction hypothesis we have that

q(G1) ≥ α(G1) − 1 + Γ(α(G1), τ(G1)),

q(G2) ≥ α(G2) − 1 + Γ(α(G2), τ(G2)).

Using the above formulas and Lemma 2.1(iv) we have that

q(G) = q(G1) + q(G2) + 1≥ α(G1) − 1 + α(G2) − 1 + Γ(α(G1), τ(G1)) + Γ(α(G2), τ(G2)) + 1= α(G) − 1 + Γ(α(G1), τ(G1)) + Γ(α(G2), τ(G2))(iv)≥ α(G) − 1 + Γ(α(G), τ(G))

Note that α(G) = α(G1) + α(G2) and τ(G) = τ(G1) + τ(G2).

Case 3 Assume that there does not exist a bridge satisfying the aboveconditions, that is, for all the bridges of G we have that τ(G1) = 0 orτ(G2) = 0. In this case we must have that G is equal to an α-criticalgraph G1 with a vertex of G1 being the center of a star K1,l. Moreoverwe have that τ(G) = τ(G1) and α(G) = l+α(G1) because G1 is vertex-critical and therefore each vertex belongs to a minimum vertex cover.Now using Case 1 and Lemma 2.1(ii), we obtain,

q(G) = l + q(G1) ≥ l + (α(G1) − 1 + Γ(α(G1), τ(G1)))

= α(G) − 1 + Γ(α(G1), τ(G))(ii)≥ α(G) − 1 + Γ(α(G), τ(G)).

3 A classification of q-minimal graphs

A 1-linking of a graph G is a new graph G′ with the same vertex setas G but obtained from G by adding the minimum number of edgespossible such that G′ be connected. The graph G is called the subjacentgraph of the 1-linking graph G′ and the edges that we add are calledthe linking edges.

Clearly a 1-linking graph G′ of a disconnected graph G can be ob-tained by adding c(G)−1 edges, where c(G) is the number of connectedcomponents of G. This definition is equivalent to the one given in [1] ofa tree-linking of a graph.


A graph G is a Turan graph, denoted by T (a, t), if G is the disjointunion of a − s complete graphs with r vertices and s complete graphswith r + 1 vertices, where a + t = r(a) + s with 0 ≤ s < a.

A graph G with covering number τ(G) = t and stability numberα(G) = a is said to be a transformed Turan graph or TT graph if eitherG is isomorphic to T (a, t), or a ≤ t ≤ 2a and G can be obtained fromT (a, t) by the following construction:

Take a positive integer k such that k ≤ mink2, k3 where k2,k3

denote the number of copies in T (a, t) of K2 and K3 respectively. Forevery 1 ≤ i ≤ k replace ji copies of K2 and one copy of K3 by a cycleC2ji+3, where j1 + · · · + jk ≤ k2.

Given a 1 − linking G′ of G, we define a leaf in G′ as a connectedcomponent Gi of G incident to a unique linking edge or as a connectedcomponent Gi with the property that their exist a unique vertex v inGi such that all linking edges with one end in Gi are incident to thevertex v.

Lemma 3.1 A graph G is q-minimal if and only if G is a 1-linking ofa transformed Turan graph.

Proof: We will use double induction on the stability and covering num-ber of the graph. For α(G) = 1 we have that G must be a completegraph and the result is clear. Therefore we can assume that G is aq-minimal graph with α(G) ≥ 2.

If G is not 2-connected, using the same arguments used in cases 2and 3 in the proof of Theorem 2.3 and the induction hypothesis theresult follows readily. Hence we can assume that G is a 2-connectedgraph, in fact that, G is an α-critical graph. Therefore the proof willbe complete if we prove that G is an odd cycle.

Let G be a q-minimal and α-critical graph and let v ∈ V (G) be avertex of maximal degree.

Claim 3.2 G \ v is q-minimal.

Proof: Assume that G\v is not q-minimal, then by the same argumentsas those in case 1 in the proof of Theorem 2.3 and Lemma 2.1 we have

q(G) = q(G \ v) + deg(v) ≥ q(G \ v) + deg(v′)(v)≥ α(G) + Γ(α(G), τ(G) − 1) +

!α(G) + τ(G) − 1

α(G)

"

(iii)= α(G) + Γ(α(G), τ(G)),(2)


which is a contradiction to the q-minimality of G.

Since G is α-critical and in particular τ -critical, then α(G) = α(G \v), that is, the set of vertices N(v) must satisfy that N(v) ∩ M = ∅ forall maximum stable sets M of G \ v. Hence α(G \ N [v]) = α(G) − 1,where N [v] = N(v)∪ v. A set of vertices N in G \ v can be the set ofneighbors of v in G if and only if V (G \ v) \N induce a subgraph G′ ofG \ v with α(G′) = α(G)− 1. Moreover N is minimal under inclusion ifand only if G[V (G \ v) \ N ] is maximal under inclusion.

Now, since G \ v is q − minimal, by induction hypothesis G \ v isa 1-linking of a TT graph. In this case it is easy to find the maximalinduced subgraph G′ of G \ v with α(G′) = α(G) − 1.

Claim 3.3 Let H be a TT graph with H1, . . . ,Ha connected componentsand let L be a 1-linking of H. Take L′ to be a maximal induced subgraphof L with α(L′) = α(L) − 1, then we have that

(i) L′ is induced by the set of vertices V (L) \ V (Hi), for some Hi withα(Hi) = 1, or

(ii) L′ is induced by the set of vertices in V (L) \ v1, v2, v3, wherev1, v2, v3 are vertices of an odd cycle Hj such that Hj\v1, v2, v3is a disjoint union of paths with an even number of vertices, or

(iii) L′ satisfies the following conditions: (1 ) V (Hi) ∩ V (L′) = ∅ forall Hi, (2 ) if Hi is an odd cycle, then V (Hi) ⊂ V (L′), (3 ) ifHi is a complete graph such that V (Hi) ⊂ V (L′), then for allv ∈ V (Hi)∩V (L′) there exist at least one linking edge ev incidentto v.

Proof: If V (L′) ∩ V (Hi) = ∅ for some 1 ≤ i ≤ a with α(Hi) = 1, thenL′ = L[V (H) \ V (Hi)], since V (L′) ⊆ V (H) \ V (Hi) and α(L[V (H) \V (Hi)]) = α(L) − 1.

Therefore we can assume that if L′ = L \ Hi with α(Hi) = 1, thenV (Hi) ∩ V (L′) = ∅ for all Hi with α(Hi) = 1.

If Hj is an odd cycle with 2m + 1 vertices and since all the properinduced graphs of a cycle are paths Pn with α(Pn) = ⌈n

2 ⌉, then α(Hj \C) = α(Hj) − 1 for some C ⊂ V (Hj) if and only if Hj [Cc] is a disjointunion of three paths Pm1 , Pm2 , Pm3 with m1,m2,m3 ≥ 0 even numbersand such that m1+m2+m3 = 2(m−1). Therefore, either L′ is describedas in (ii) or V (Hj) ⊂ V (L′) for all Hj with α(Hj) ≥ 2.

To finish, if L′ is not given by (i) or (ii), then we can assume thatV (L′)∩V (Hi) = ∅ for all 1 ≤ i ≤ a, moreover if Hj is an odd cycle then


V (Hj) ⊂ V (L′). Clearly if v ∈ V (Hi)∩V (L′) and v is not incident to anylinking edge, then V (Hi) ⊂ V (L′) because α(L′) = α(L[V (L′)∪V (Hi)])

Applying Claim 3.3 to G \ v it is easy to conclude that

• G is a complete graph whenever G \ N [v] is as in (i).

• G is not q-minimal whenever G \ N [v] is as in (ii).

Therefore it only remains to considerer when G \ N [v] satisfies theconditions given in case (iii). Let Hi0 be a complete graph such thatHi0 is a connected component of the subjacent graph of G \ v (a TTgraph) with V (Hi0) ⊂ V (G′) (note that by Claim 3.3 (iii) there existsat least one graph Hi with this condition) and take P = V (Hi0)∩V (G′)and Q = V (Hi0) \P . Since G \ v is q-minimal, then we have that for allu ∈ P , (G \ v) \ u in not connected. For all u ∈ P , let Gu the union ofthe connected components of (G \ v) \u such that V (Gu)∩V (Hi0) = ∅.Note that Gu is an induced subgraph (a disjoint union of 1-linking ofTT graphs) of G \ v such that Gu is joined to u by linking edges. Notethat if |V (Hi0)| ≥ 2, then Gu is unique.

Here we need to considerer two cases, the first case is when Gu isnot a TT graph. If S is a leaf of Gu not joined to u by a linking edge,then by the 2-connectivity of G we have that v must be incident withat least one vertex of S. In the other case, if Gu is a TT graph, thenby the 2-connectivity of G we have that there exists at least one vertexw such that w is incident to v and we can consider that S = Gu is theonly leaf of Gu.

r s

Gr

GsHi0

S1

S2P

QG \ v

Moreover, by Claim 3.3 (iii) we have that if vs is the unique vertexof S such that all the linking edges with one end in S are incident to vs,then v must be incident with all the vertices of S \ vs, more precisely


we have that:

deg(v) ≥ |Q| +!

u∈P

!

Hj∈L(Gu)

(|Hj |− 1)(∗)≥ |Hi0 |,(3)

where L(Gu) is either the set of leaves of Gu not joined to u when Gu

is not a TT graph or equal to Gu when Gu is a TT graph. We haveequality in (∗) if and only if all the leaves of Gu are isomorphic to K2

and if Gu is not a TT graph, then Gu has exactly two leaves.

Now, let Hi0 with |Hi0 | = k ="|V (G \ v)|

α(G)

#, using that

deg(v) = q(G) − q(G \ v)

≤ Γ(α(G), τ(G)) − Γ(α(G), τ(G) − 1)

=$|V (G)|α(G)

%− 1

≤ k

and deg(v) ≥ 2k − 2 (by Equation (3)), then k must be equal to 2. Asimilar argument shows that k = 2 when we take |Hi0 | = k =

&|V (G\v)|

α(G)

'.

Hence Hi0 = K2, deg(v) = 2,&|V (G\v)|

α(G)

'≤ 3, |P | = 1 and Gu has only

two leaves. Therefore G must be an odd cycle since Gu must be a1− linking of a TT graph whose components are all isomorphic to K2.

4 A relation between the stability and coveringnumber

In this section we present some relations between two important in-variants of a graph G, the stability number α(G) and covering numberτ(G).

The origin of our interest in the study of these relations comes frommonomial algebras, more precisely we have that: the stability numberα(G) of a graph G, is equal to the dimension of the Stanley-Reisnerring associated to the graph G; and the covering number τ(G) of Gis equal to the height of the ideal associated to the graph G. Finally,α(G) − δ(G) is an upper bound to the depth of this ring.


From the algebraic point of view an important class of rings is givenby those rings R such that their dimension is equal to their depth. Therings in this class are called Cohen-Macaulay rings.

A graph is Cohen-Macaulay if the Stanley-Reisner ring associated toit, is Cohen-Macaulay. We have that if a graph G is Cohen-Macaulay,then δ(G) = 0; note that this is a necessary condition but not a sufficientcondition.

The family of graphs with δ(G) ≥ 1 correspond to the Stanley-Reisner rings that have a large depth. Moreover, the dimension minusthe depth is bounded below by δ(G), and hence δ(G) is a measure ofhow far these rings are from being Cohen-Macaulay.

The following results are in the spirit of [3], in that paper the authorswhere motivated in bounding invariants for edge rings. In this paper weconcentrate mainly in the combinatorial aspects of these bounds.

The theorem below, gives an idea of the class of graphs that areCohen-Macaulay and of those graphs that are far from being Cohen-Macaulay. We thank N. Alon (private communication) for some usefulsuggestions in making the proof of this result simpler and more readable.

Theorem 4.1 Let G be a graph without isolated vertices, then

α(G) ≤ τ(G)[1 + δ(G)].

Proof: First, fix a minimal vertex cover C with τ(G) vertices. Let v ∈ C,then there exist a maximal stable set M ′ with v ∈ M ′ and |M ′| ≥ σv(G).Hence there exist a natural number k ≤ τ(G) and T1, . . . , Tk maximalstable sets with |Ti| ≥ σv(G) such that

C ⊂k!

i=1

Ti.

Let M = V \ C and take Ci = C ∩ Ti and Mi = M ∩ Ti for alli = 1, . . . , k. Since the graph G does not have isolated vertices, thenfor all v ∈ M there exists an edge e of G with e = v, v′. Now, asC = V (G) \ M and C is a vertex cover we have that v′ ∈ C, that is

(4) M =k!

i=1

(M ∩ N(Ci)).


Since Si = V (G)\Ti = (C \Ci)∪ (M \Mi) is a minimal vertex coverwith |Si| ≤ n − σv(G) for all i = 1, . . . , k, then

|C \ Ci| + |M \ Mi| = |(C \ Ci) ∪ (M \ Mi)| = |Si| ≤ n − σv(G).

Hence as M ∩ N(Ci) ⊆ M \ Mi we have that

(5)

|M ∩ N(Ci)| ≤ |M \ Mi| ≤ n − σv(G) − |C \ Ci|= |C| + α(G) − σv(G) − |C \ Ci|= |Ci| + α(G) − σv(G)= |Ci| + δ(G).

Taking

Ai = Ci \ (i−1!

j=1

Cj) and Bi = (M ∩ N(Ci)) \i−1!

j=1

(M ∩ N(Cj)),

we have that

(6) |Ci \ Ai| ≤ |M ∩ N(Ci \ Ai)|,

since if |Ci \Ai| > |M ∩N(Ci \Ai)|, then C \(Ci \Ai)∪(M ∩N(Ci \Ai))would be a vertex cover of cardinality |C \(Ci \Ai)|+ |M ∩N(C \Ai)| <|C|; a contradiction.

To finish the proof, we use the inequalities (5) and (6) to concludethat

|Bi| = |(M ∩ N(Ci))|− |(M ∩ N(Ci)) ∩"i−1

j=1(M ∩ N(Cj))|= |(M ∩ N(Ci))|− |M ∩ N(Ci) ∩ N(

"i−1j=1 Cj))|

(5)≤ |Ci| + α(G) − σv(G) − |M ∩ N(Ci ∩

"i−1j=1 Cj))|

(6)≤ |Ci| + α(G) − σv(G) − |Ci \ Ai|

= |Ai| + α(G) − σv(G) = |Ai| + δ(G).

Therefore

α(G)(4)= |

"ki=1(M ∩ N(Ci))| =

#ki=1 |Bi| ≤

#ki=1(|Ai| + δ(G))

≤ |C| + τ(G)δ(G)

= τ(G)[1 + δ(G)]


Remark 4.2 If δ(G) > 0, then we have that α(G) = τ(G)[1 + δ(G)]if and only if G is formed by a clique Kτ(G) with each vertex of thisclique being the center of a star K1,δ(G)+1. Furthermore, if δ(G) = 0and α(G) = τ(G), then the graph has a perfect matching.

Kτ(G)

K1,δ(G)+1

Figure 1: The graph formed by a clique Kτ(G) with each vertex of thisclique being the center of a star K1,δ(G)+1.

Let

αcore(G) =stable set!

|Mi|=α(G)

Mi and τcore(G) =vertex cover!

|Ci|=τ(G)

Ci,

be the intersection of all the maximum stable sets and of all theminimum vertex covers of G, respectively.

A graph is τ -critical if τ(G\v) < τ(G) for all the vertices v ∈ V (G),that is, a graph is τ -critical if and only if αcore(G) = ∅. Similarly,we have that G is a B-graph if and only if τcore(G) = ∅. We defineBα∩τ = V (G) \ αcore(G) ∪ τcore(G).

Proposition 4.3 Let G be a graph, then

V (G) = αcore(G) $ τcore(G) $ Bα∩τ ,

furthermore

(i) N(αcore(G)) ⊆ τcore(G),


(ii) G[αcore(G)] is a trivial graph,

(iii) G[Bα∩τ ] is both a τ -critical graph as well as a B-graph withoutisolated vertices.

Proof: Clearly αcore(G) ∩ τcore(G) = ∅. Now, since G[V (G) \ τcore(G)]is a B-graph, we have that, αcore(G) ⊂ V (G) \ τcore(G) is the set of iso-lated vertices of G[V (G) \ τcore(G)]. Therefore N(αcore(G)) ⊆ τcore(G),proving (i). Hence, we have that G[αcore(G)] is a graph without edges,giving (ii). Finally, by definition of Bα∩τ we obtain (iii).

Example 4.4 To illustrate the previous result consider the followinggraph:

v1

v2

v3

v4

v5v6 v7

since α(G) = 3, τ(G) = 4 and v3, v4, v5, v3, v4, v6, v3, v4, v7 arethe maximum stable sets of G, we have that

• αcore(G) = v3, v4,

• τcore(G)] = v1, v2,

• Bα∩τ = v5, v6, v7.

Remark 4.5 It is easy to see that if v is an isolated vertex, thenv ∈ αcore(G), in a similar way we have that if deg(v) > τ(G), thenv does not belong to any stable set with α(G) vertices and thereforev ∈ τcore(G). Note that in general the induced graph G[Bα∩τ ] is notnecessarily connected.

Corollary 4.6 Let G be a graph, then

α(G) − |αcore(G)| ≤ τ(G) − |τcore(G)|.


Proof: By Proposition 4.3 we have that G[Bα∩τ ] is a B-graph. Now,since

α(G[Bα∩τ ]) = α(G) − |αcore(G)| and τ(G[Bα∩τ ]) = τ(G) − |τcore(G)|,

and by applying Theorem 4.1 to G[Bα∩τ ] we obtain that

α(G) − |αcore(G)| ≤ τ(G) − |τcore(G)|.

Remark 4.7 The bound of Corollary 4.6 improves the bound given in[5, Theorem 2.11] for the number of vertices in αcore(G). Their resultstates:

If G is a graph of order n and

α(G) > (n + k − min1, |N(αcore(G))|)/2,

for some k ≥ 1, then |αcore(G)| ≥ k + 1. Moreover, if

(n + k − min1, |N(αcore(G))|)/2

is even, then |αcore(G)| ≥ k + 2.

Notice that if α(G) ≥ n/2 + k′/2, our bound gives,

|αcore(G)| ≥ k′ + |τcore(G)|.

Remark 4.8 After this paper was submitted, the authors learned thatTheorem 2.3 was also obtained independently in [2].

Isidoro GitlerDepartmento de Matematicas,Centro de Investigacion y de Estu-dios Avanzados del IPN,Apartado Postal 14–740,07000 Mexico City, D.F.,[email protected]

Carlos E. ValenciaDepartmento de Matematicas,Centro de Investigacion y de Estu-dios Avanzados del IPN,Apartado Postal 14–740,07000 Mexico City, D.F.,[email protected]

References

[1] Bougard N.; Gwenael Joret, Turan’s theorem and k-connectedgraphs, manuscript.

[2] Christophe J. et al, Linear Inequalities among Graph Invariants:using GraPHedron to uncover optimal relationships, e-print availableon Optimization Online.


[3] Gitler I.; Valencia C. E, Bounds for invariants of edge-rings, Comm.Algebra 33 (2005), 1603–1616.

[4] Gitler I.; Valencia C. E., Bounds for graph invariants, arXiv:math.CO/0510387.

[5] Levit V. E.; Mandrescu E., Combinatorial properties of the family ofmaximum stable sets of a graph, Discrete Appl. Math. 117 (2002),149–161.

[6] Ore O., Theory of graphs, American Mathematical Society Collo-quium Publications, Vol. XXXVIII, American Math. Society, Prov-idence, R. I., 1962.

[7] Turan P., Eine Extremalaufgabe aus der Graphentheorie, Mat. Fiz.Lapok 48 (1941), 436–452.

Morfismos, Comunicaciones Estudiantiles del Departamento de Matematicas delCINVESTAV, se termino de imprimir en el mes de junio de 2007 en el taller dereproduccion del mismo departamento localizado en Av. IPN 2508, Col. San PedroZacatenco, Mexico, D.F. 07300. El tiraje en papel opalina importada de 36 kilo-gramos de 34 × 25.5 cm consta de 500 ejemplares con pasta tintoreto color verde.

Apoyo tecnico: Omar Hernandez Orozco.

Contenido

A unified approach to continuous-time discounted Markov control processes

Tomas Prieto-Rumeau and Onesimo Hernandez-Lerma . . . . . . . . . . . . . . . . . . . 1

On bounds for the stability number of graphs

Isidoro Gitler and Carlos E. Valencia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Documents

Morfismos, Vol 10, No 1, 2006