IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 5, NO. 7, JULY 2006 1651

Transactions Papers

A Learning Approach for Prioritized HandoffChannel Allocation in Mobile Multimedia Networks

El-Sayed El-Alfy, Member, IEEE, Yu-Dong Yao, Senior Member, IEEE, and Harry Heffes, Fellow, IEEE

Abstract— An efficient channel allocation policy that priori-tizes handoffs is an indispensable ingredient in future cellularnetworks in order to support multimedia traffic while ensuringquality of service requirements (QoS). In this paper we studythe application of a reinforcement-learning algorithm to developan alternative channel allocation scheme in mobile cellular net-works that supports multiple heterogeneous traffic classes. Theproposed scheme prioritizes handoff call requests over new callsand provides differentiated services for different traffic classeswith diverse characteristics and quality of service requirements.Furthermore, it is asymptotically optimal, computationally inex-pensive, model-free, and can adapt to changing traffic conditions.Simulations are provided to compare the effectiveness of theproposed algorithm with other known resource-sharing policiessuch as complete sharing and reservation policies.

Index Terms— Channel allocation, cellular multimedia net-works, handoffs, Markov decision processes, dynamic program-ming, reinforcement learning.

I. INTRODUCTION

ADVANCES in wireless mobile networks have fueled therapid growth in research and development to support

broadband multimedia traffic and user mobility. Since theavailable radio resources are scarce, an efficient allocationpolicy is an essential ingredient in order to accommodatethe increasing demands for wireless access. Other motivationsinclude the increasing need to support broadband multimediatraffic and user mobility as well as to ensure quality ofservice (QoS) requirements in a complex dynamic environ-ment. Issues that make this problem more complicated arethe traffic heterogeneity, i.e., diverse traffic characteristicsand QoS requirements; as well as the time-varying natureof the network traffic. To increase the allocation efficiencyof radio resources, at the network planning level, there is amigration from macro-cells to micro-cells and even pico-cells.As a result a substantial portion of traffic in each cell results

Manuscript received February 1, 2001; accepted June 22, 2002. Theassociate editor coordinating the review of this paper and approving it forpublication was S. Tekinay.

E.-S. El-Alfy is with the Computer Science Department, King Fahd Uni-versity of Petroleum and Minerals, Saudi Arabia (email: alfy@kfupm.edu.sa).

Y.-D. Yao and H. Heffes are with the Wireless Information SystemsEngineering Laboratory (WISELAB), Department of Electrical and ComputerEngineering, Stevens Institute of Technology, Hoboken, NJ 07030 USA(email: yyao@stevens.edu; hheffes@stevens.edu).

Digital Object Identifier 10.1109/TWC.2006.01023

from calls migrating from one cell to another (handoffs).However, due to the scarcity of the available resources andthe co-channel interference, there is a possibility that a callin progress may be dropped due to handoff failure because ofnetwork congestion.

Two important connection-level performance metrics arenew call blocking probability and call dropping probability.We consider call dropping as a result of handoff failure due toinsufficient resources after initiating a handoff request. Thesetwo metrics are always conflicting. From a user point of view,dropping a call in progress due to a handoff failure is moreundesirable than rejecting a new call. Therefore, an optimal al-location policy that prioritizes handoff requests over new callsis required to maximize the network throughput and guaranteethe quality of service for both active and new calls/sessions. Inthis paper the QoS is considered to be a function of blockedcalls and handoff failures. Multimedia traffic is defined interms of traffic characteristics and resource requirements,e.g. bandwidth requirements. Resource allocation has beenwidely studied in several disciplines including engineering,computer science, and operation research. In cellular networks,the efficient use of radio channels is affected by severalfactors including power control, user mobility pattern, handoffinitiation process, and handoff scheduling policy. Two relatedschemes are the adopted channel assignment scheme amongcells and the channel allocation strategy for different trafficclasses, i.e., admission control policy.

Several schemes have been proposed in the literature forchannel assignment ranging from complete partitioning, e.g.,fixed channel assignment (FCA) to complete sharing, e.g.,dynamic channel assignment (DCA) and their extensions [1],[2] and references therein. Recently reinforcement learning[3], [4] has been applied to this problem [5], [6]. In theseschemes, a call is admitted, with no differentiation betweennew calls and handoffs, if a channel is available such thatthe channel reuse constraint is not violated. Since rejectinga new call is less serious than dropping an ongoing call,a tremendous research effort has been devoted to allocationpolicies that prioritize channel assignment to handoff requestswithout significantly increasing the new call blocking rates[2], [7]–[14].

One well-known method that prioritizes handoff over new

1536-1276/06$20.00 c© 2006 IEEE

1652 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 5, NO. 7, JULY 2006

call requests is the guard channel, cutoff priority, guardthreshold, or trunk reservation policy. The basic idea is toa priori reserve certain number of channels in each cell tohandle handoff requests besides allowing the handoff requeststo first compete with the new calls for the remaining channels.However since reducing the handoff failures cannot be attainedwithout sacrificing the blocking probability of new calls,the adopted reservation policy has to answer two questions:how many channels should be reserved in each cell andwhether they should be a priori reserved [2], [7]–[9] ordynamically changed to adapt to traffic conditions [10], [11].For a single traffic class, the optimal policy that minimizesa weighted linear function of blocking probabilities has asimple structure of threshold type [12]. The optimal thresholdcan be determined using analytical models [13] and thetheory of Markov decision processes (MDP) [15], [16], e.g.,dynamic programming [12], [13], [17] or linear programming[14]. For multiple traffic classes, there is a class of policiesknown as coordinate convex policies [18]. However, undercertain performance indices, the optimal policy is no longerof coordinate convex type [18], [19]. MDP approach can beused to formulate such problem for which the optimal policysearch can be performed via dynamic programming or linearprogramming techniques [20]. All these methods are off-line,static and are based on the assumption that traffic parametersare not changing over time. Furthermore, they rely on theassumption of the knowledge of a perfect analytical statetransition and cost model which is not easy to find espe-cially in a complex dynamically changing environment. Forsuch dynamically changing networks supporting several trafficclasses with time varying characteristics and requirements, theproblem complexity is substantially high, i.e., the state-actionspace becomes exponentially growing, and exact solutionsusing traditional optimization techniques become intractable.This is known in literature as the “Curse of Dimensionality"[17]. In this paper we study the application of an averagecost reinforcement learning (RL) methodology [21] to developa channel allocation policy that prioritizes the admission ofhandoff requests over new call requests. The performance ofthe proposed algorithm is compared with the optimal guardchannel and complete sharing policies. The RL scheme is astochastic approximation to dynamic programming algorithms(DP) and can be implemented as an on-line “allocation policy"that learns from direct interactions with the network. Also,it has a number of other advantages: simple implementation,model-free, since there is no need for prior knowledge orestimation of the network dynamics, and self-adjusting totime varying traffic conditions. The remainder of this paperis organized as follows. The next section describes the trafficmodel and the performance measures used; and formallydefines the optimization problem to be solved. Section 3presents the reinforcement learning solution. We first introducebriefly the average cost semi-Markov decision process andthen develop an allocation policy based on reinforcementleaning approach for multiple traffic classes and for a singletraffic class. Simulations and numerical results are given inSection 4 for two scenarios. In the first scenario we simulatethe learning scheme for multiple traffic classes. A comparisonwith complete sharing policy, both analytical and simulation,

are also given. In the second scenario we consider a singletraffic class for which we give the analytical exact optimalsolution. Then, we run the discrete event simulator for ourlearning scheme and compare its performance with the cor-responding results for complete sharing and optimal guardpolicies. Finally, in Section 5 we present conclusions andfuture work.

II. TRAFFIC MODEL AND PROBLEM DESCRIPTION

A. Traffic Model

Consider a cellular network with a fixed number of channels(or bandwidth units) where the service region is divided intosmall cells with a general spatial layout. Here the conceptof channel is used in a generic sense independent of theaccess technology used whether frequency division multipleaccess (FDMA), time division multiple access (TDMA), orcode division multiple access (CDMA). Multimedia traffic isdefined in terms of their traffic characteristics and resourcerequirements, e.g. bandwidth requirements. We consider Kdifferent classes indexed by k = 1, 2, . . . ,K where eachclass is characterized by a set of traffic parameters: meanarrival rate, bandwidth requirement (number of channels),mean service time, and a weighting factor indicating therelative importance of each class or its priority level.

There are two sources of type-k traffic in each cell based onthe call originating location: new call arrivals, i.e., calls orig-inated within the cell, and handoff request arrivals, i.e., callsmigrating from neighboring cells into that given cell. Underthe assumption of spatial uniform traffic conditions and fixedchannel assignment strategy between cells, the traffic model ofa cellular system can be decomposed into independent cells orclusters [14]. Consider a typical cell with C channels; then thetraffic within that cell can be modeled using a C-server systemwhich is an extension of Erlang’s loss model [18], [19]. Fig. 1(a) shows a particular cell within a cellular system and Fig. 1(b) shows an equivalent heterogeneous channel traffic modelin which the C-channels, available in the cell, are representedas C-servers. There are two arrival streams for each class: onerepresenting an aggregate traffic stream for handoff requestsinto the cell with mean arrival rate λHk and the other streamrepresents the new call arrival within the cell with mean arrivalrate λNk. The arrival rate matrix is given by,

Λ =[λ1 λ2 · · · λk · · · λ2K

], (1)

where

λk =

{λNk, for 1 ≤ k ≤ K

λH(k−K), for K + 1 ≤ k ≤ 2K

For non-prioritizing techniques such as complete sharingapproach, there is no difference between new calls and hand-offs. Therefore, the arrival rate vector is given by

Λ =[λ1 λ2 · · · λk · · · λK

], (2)

whereλk = λNk + λHk, ∀k ∈ {1, 2, . . . ,K}

Each call of type-k, whether a new call or handoff, requestsa fixed number of channels given by bk. These channels

EL-ALFY et al.: A LEARNING APPROACH FOR PRIORITIZED HANDOFF CHANNEL ALLOCATION IN MOBILE MULTIMEDIA NETWORKS 1653

(a)

Dropped calls

Handoff arrivals

New call arrivals

Handoffs out ofthe cell

Blocked calls

Successfulterminations

Available

Occupied

(b)

Fig. 1. Cellular system layout and traffic model.

are released simultaneously upon the call departure (whethercompletion or handoff out of the cell). In multimedia networksthis may corresponds to a constant bit rate (CBR) service type.However for a variable bit rate (VBR) service class this mayrepresent the peak value or the effective value [22] of therequested bandwidth. The requested band vector is,

b =[b1 b2 · · · bk · · · bK

](3)

Calls leave the cell (the queueing system) as a result ofsuccessful call completion or as a result of handoff to one ofthe neighboring cells. Since the system has finite capacity anew call request or handoff request may be rejected. Withoutloss of generality of the algorithm, we assume that blockedcalls are cleared from the system [18]. We assume that the ar-rivals of class-k new call and handoff requests are according tomutually independent Poisson processes. We also assume thatclass-k call duration and the cell dwelling time, i.e., the timeuntil handing-off out of the cell, TDk and THk respectively,are mutually independent and exponentially distributed withmeans and respectively. TDk and THk are also independent thearrival processes and other traffic classes’ times. The channelholding time for class-k, Tk, is the minimum of the call

duration and time until handoff and can be expressed as

Tk = min (TDk, THk) (4)

The channel holding time PDF is given by

FTk(t) = Pr {Tk ≤ t} = 1 − Pr {TDk > t, THk > t}

= 1 − e−(μDk+μHk)t(5)

Thus, the channel holding time for class-k traffic is alsoexponential with mean given by,

1μk

=1

μDk + μHk(6)

So, the channel holding time vector is expressed as

T =[μ−1

1 μ−12 · · · μ−1

k · · · μ−1K

](7)

B. Coordinate Convex Policies

The network state, n, can be defined by the vector

n =[n1 n2 · · · nk · · · nK

], (8)

where nk is the number of active calls of type-k traffic. Thesystem state space is a finite set given by

S =

{n ∈ {0, 1, . . . , C}K |n · bT =

K∑i=1

nibi ≤ C

}(9)

Under a class of policies; known as coordinate convex poli-cies [18], [20], such as complete sharing, complete partitioningand subsets, the admission policy is completely specified witha restricted set of states Σ ⊆ S. For example, in completesharing, a request is admitted if the requested number ofchannels is available regardless of the request type. Therefore,for complete sharing Σ = S. The natural evolution of thestochastic process {n(t), t ≥ 0} represents a multidimensionalcontinuous-time Markov chain.

For coordinate convex policies, the admission rule is simpleand depends on the next state of the system. Let n+

k representthe next state of the network when a new call or handoff oftype-k has arrived and has been accepted given that the currentstate is n, i.e.,

n+k =

[n1 n2 · · · nk + 1 · · · nK

](10)

The admission decision depends on what the next state ofthe system would be if the requested band is granted, and isgiven by

βNk(n) = βNk(n) =

{1, if n+

k ∈ Σ0, elsewhere

(11)

where 1 means accept and 0 means reject. This means that theadmission rule is the same for both new call and handoff trafficof type-k. To provide preferential treatment for handoffs,the state definition needs to incorporate different variablesfor new calls and handoffs of each type as if they are twodifferent classes. But this leads to doubling the state spacewhich increases the complexity of any solution procedure. For

1654 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 5, NO. 7, JULY 2006

coordinate convex policies, the transition rate from state x tostate y is,

rxy =

⎧⎪⎨⎪⎩λk(x), if x = n ∈ Σ and y = n+

k ∈ Σxkμk, if x = n ∈ Σ and y = n−

k ∈ Σ0, elsewhere

(12)

where λk(x) = λNkβNk(x) + λHkβHk(x) and n−k =

[ n1 n2 . . . nk − 1 . . . nK ].Following [19], according to the law of conservation of

flow, the equilibrium balance equations of any policy can beexpressed as:[

K∑k=1

(λk(n) + nkμk)

]p(n) =

K∑k=1

(nk + 1)μkp(n+k )

+K∑k=1

λk(n−k )p(n−

k ),

∑n∈ Σ

p(n) = 1

(13)

where p(n) is the statistical equilibrium time-average proba-bility that the system occupies state n. This set of equationscannot in general be solved by recurrence to find closedform solutions for the equilibrium state probabilities. How-ever, under coordinate convex policies, the equilibrium statedistribution has product form and expressed as [19]:

p {n = (n1, n2, . . . , nk, . . . , nK)} =K∏k=1

ρnk

k

nk!G−1(Σ) (14)

where,

G(Σ) =∑

n∈ Σ

[K∏k=1

ρnk

k

nk!

], ρk =

λkμk

These results are still applicable even for arbitrary channelholding time distributions due to the insensitivity property[18]. In a most general class of policies, the admission rulemay deny service, based on the type of the traffic, even ifthe requested band is available. In such cases, βNk(n) andβHk(n) may be different and they represent the probability ofadmitting a new call or a handoff request of type-k respectivelywhen the system is currently in state n. The new call andhandoff blocking probabilities experienced by each traffic typecan be expressed as,

BNk = 1 −∑n∈S

βNk(n)p(n)

BHk = 1 −∑n∈S

βHk(n)p(n)(15)

C. The Optimization Problem

The objective of this study is to find a channel allocationpolicy that minimizes a weighted linear function of new calland handoff blocking probabilities of each type as defined by,

P =K∑k=1

{wNk

λNkλNk + λHk

BNk + wHkλHk

λNk + λHkBHk

}(16)

Call departure

(successfulcompletionor handoff)

state

Traffic lineControl line

Valuefunction

Admission policy

Immediate cost

New call arrivals

Blockednew call

Blockedhandoffs

New calls admitted

RL Allocation Controller

Handoff arrivalsHandoffs admitted

Fig. 2. Traffic model embedded with call admission controller.

sn-1 snan-1

cn-1

n-1

s2s0=sa0

c0

0

s1 s3a2

c2

2

a1

c1

1

...

Fig. 3. A sample path given the starting state is s and following a givenpolicy π, where at is the action selected at decision epoch t.

where wNk and wHk represent the relative weights or thecosts incurred when rejecting a new call or handoff of type-k.wHk > wNk reflects the fact that rejecting a handoff is moreundesirable than blocking a new call. The aim of any admis-sion control strategy is to determine the parameters, βNk(n)and βHk(n), for all states, n ∈ S. Since the optimal policymay not be coordinate convex, we need to look at a broad classof control policies. However, as pointed in [20], the productform solution does not exist for the equilibrium balance equa-tions outside the class of coordinate convex policies. Anotherclass of algorithms, based on semi-Markov decision problems(SMDP) [15], [16], allows the optimization over all policies.If a perfect system model is known, the optimal solution ofSMDP can be attained via dynamic programming (DP) orlinear programming (LP) techniques for reasonable problemsize. However due to the high dimensionality of the problem,the computational complexity of these traditional approachesis prohibitive. Reinforcement learning, a class of successiveapproximation methods, can be used to approximately solvethe SMDP incrementally on-line without a priori knowledgeof the system model. The RL allocation controller learns anadmission policy from direct interaction with the network asdepicted in Fig. 2. In the next section we present a class ofRL channel allocation schemes.

III. CHANNEL ALLOCATION POLICY USING

REINFORCEMENT LEARNING

In this section, we briefly review the semi-Markov deci-sion problems (SMDP) under the average cost criterion [15],[16]. Then, the prioritized handoff channel allocation problemamong different traffic classes is formulated as an infinite-horizon finite-state SMDP under the average cost criterion.Finally, a class of model-free schemes based on reinforcementlearning is used to obtain an asymptotically optimal allocationprocedure.

EL-ALFY et al.: A LEARNING APPROACH FOR PRIORITIZED HANDOFF CHANNEL ALLOCATION IN MOBILE MULTIMEDIA NETWORKS 1655

A. Average Cost Semi-Markov Decision Problem

An SMDP is defined by the following three components:a dynamic system, an immediate cost (reward) function, andan objective function. The dynamic system is modeled as acontrolled Markov chain. At random points in time (decisionepochs) the controller observes the system state and selects anaction. As a result the system state will change as a functionof the current state, selected action, and external disturbance.Also, the controller receives an evaluative feedback (reinforce-ment) signal to reflect the immediate (stage) cost incurred.The controller maintains an evaluative (objective) function foreach state (state value function), or for each state-action pair(action value function), which is usually an accumulation ofthe immediate costs incurred over time. There are three mostcommon definitions for the value functions: Finite-horizonexpected total cost, infinite-horizon expected discounted totalcost, and expected average cost. In the following we usean expected average cost criterion. A deterministic stationarypolicy is a mapping from states to actions, i.e., π : S → Awhere S is a finite set of all states and A is a finite set ofall actions. Assume the system starts at state s0 = s andfollowing policy π, then the long-run average cost, or policygain, is given by

gπ(s) = limn→∞

E{∑n−1

t=0 cπt (s)

}E{∑n−1

t=0 τt

} (17)

where cπt (s) is the cost incurred at decision time t given thatthe starting state is s and the implementation policy is π. Atypical scenario for the evolution of the embedded Markovchain is illustrated in Fig. 3.

For ergodic Markov decision processes, the average costexists and is independent of the starting state [16], i.e.,gπ(x) = gπ(y) = gπ ∀x, y ∈ S and ∀π ∈ Π.

Here Π is the space of feasible policies and gπ is the long-term average cost starting from any initial state and followingpolicy π. The controller objective is to determine a policy inorder to minimize the long-run average cost. Formally, we areseeking a policy π∗ with corresponding long-run average costwhich is optimal, i.e., g∗ = min

π∈Πgπ .

According to Bellman’s optimality principle [15], for finitestate and action spaces, an optimal policy has the propertythat, whatever the initial state and decision are, the remainingdecisions must form an optimal policy with regard to the re-sulting state from the first transition. The optimality equationsfor the average cost semi-Markov decision process have thefollowing recursive form:

h∗(x) = mina∈Ai

⎧⎨⎩c(x, a) − g∗τ(x, a) +

∑y∈S

p(y|x, a)h∗(y)⎫⎬⎭ ,

∀x ∈ S(18)

where h∗(x) is an optimal average-adjusted state value func-tion h∗ : S → �, c(x, a) is the expected immediate costincurred when being in state s and action a is selected; andτ(x, a) is the average sojourn time until the next decisionepoch when being in state s and action a is selected. The state

transition probability p(y|x, a) is defined as the probabilitythat the state at the next decision epoch is y given that thesystem currently in state x and performing action a. Solvingthese equations results in the optimal values g∗ and h∗(x)∀x ∈S. Knowing h∗(x)∀x ∈ S, the corresponding optimal policyis determined by

π∗(x) = arg mina∈Ai

{c(x, a) − g∗τ(x, a)

+∑y∈S

p(y|x, a)h∗(y)}, ∀x ∈ S(19)

Another approach is to define a value function for eachavailable action in each state, or Q-values [23], as follows

Q∗(x, a) = c(x, a) − g∗τ(x, a) +∑y∈S

p(y|x, a)

× minb∈Ay

{Q∗(y, b)}, ∀x ∈ S, x ∈ Ax(20)

where h∗(x) = mina∈Ax

Q∗(x, a), ∀a ∈ S.

The optimal action in each state is determined according to

π∗(x) = arg mina∈Ax

Q∗(x, a) (21)

B. Reinforcement Learning Schemes

The computational complexity hinders the classical opti-mization methods for solving the optimality equations, suchas linear programming and dynamic programming, to scalewell for practical dynamic networks. Reinforcement learningcan be used to learn an optimal policy on-line from directinteractions with the environment by successively approxi-mating dynamic programming algorithms. RL is a class ofalgorithms that are capable of improving their performanceincrementally. There are two most common techniques forsolving reinforcement learning problems: Sutton’s temporaldifference [24] and Watkins’ Q-learning [23].

Temporal difference learning: The temporal differencelearning scheme is a successive approximation of the asyn-chronous dynamic programming value iteration algorithm forfinding solutions of the system of equations (18). It learnsthe state value functions iteratively using the sample values,instead of the expected values, by the following incrementalaveraging rule

hnew(x) = (1−α)hold(y)+α{c(x, a, y)−gτ(x, a, y)+hold(y)}(22)

where α ∈ [0, 1] is the learning rate, or step size parameterand the setting for α may be fixed or diminishing over time.

Q-learning: The controller uses the sample information toincrementally update the state-action value function, to solvethe system of equations (20), using temporal difference asfollows

Qnew(x, a) = (1 − α)Qold(x, a) + α{c(x, a, y)−g∗τ(x, a, y) + min

b∈Ay

(Qold(y, b))} (23)

where again α ∈ [0, 1] is the learning rate. At each decisionepoch, the action selection is performed based on the estimatedQ-values in a number of ways to balance the exploration andexploitation tradeoff. One selection rule is known as the greedy

1656 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 5, NO. 7, JULY 2006

selection policy, where no exploration is made, an action isselected in state x such that

π(x) = arg mina∈Ax

{Q(x, a)} (24)

Another approach, called ε-greedy, is to select the greedyaction with a high probability and with a small probability,ε, uniformly select among other actions. The later policyhas an advantage over the greedy policy with respect tothe the rate of convergence since it allows the Q-values tobe updated for non-greedy actions and maintains a balancebetween exploiting information and exploring.

Average cost estimation: Since the controller has no infor-mation about the long-run average cost g∗ it can be similarlyestimated online using the following update rule

gnew = (1 − β)gold+

β

c(x, a, y) + minb∈Ay

Q(y, b) + mina∈Ax

Q(x, a)

τ(x, a, y)

(25)

where β ∈ [0, 1] is another learning rate. Another way toestimate g∗, using accumulated costs and times, is

g =

n−1∑i=0

c (si, ai, si+1)

n−1∑i=0

τ (si, ai, si+1)

(26)

The above estimates can update the value of g∗ at eachdecision epoch or only when selecting a greedy action.

The asymptotic convergence of reinforcement learning al-gorithms is based on the assumption that all available actionsin each state are tried infinitely often. The selection of thelearning rates and the exploration-exploitation affects the rateof convergence. For large state spaces, the Q-values canbe represented as a parameterized function using functionapproximation methods, e.g., neural networks [4].

1) Multiple Traffic Class Case: In the multiple trafficclasses with prioritized handoff problem, one obvious ap-proach to formulate the allocation problem is to treat thenew call and handoff of type-k traffic, 1 ≤ k ≤ K , as twodistinct classes. In what follows we use another more efficientformulation based on the state aggregation. In aggregatedstates we need only to use one variable for each traffic typeto indicate the number calls of that type, both new callsand handoffs, in progress. This change results in a dramaticreduction in the cardinality of the state space. Also, it allowsthe states to be visited more often and updates the valuefunctions more often. Therefore, more accurate results can beobtained. In this case, the system state at time, t, is defined as

n(t) =[n1(t) n2(t) · · · nk(t) · · · nK(t)

], (27)

where nk(t) is the number of on-going new calls of type-ktraffic at time t if 1 ≤ k ≤ K . The state space is a finite setgiven by,

S =

{n ∈ {0, 1, . . . , C}K |n · bT =

K∑k=1

nkbk ≤ C

}(28)

The decision epochs are defined to occur at discrete timescorresponding to the occurrence of external events (distur-bance). There are three external events for type-k traffic:new call arrival, handoff arrival, and call departure. Lete = [e1 e2 . . . eK . . . e2K ] be an eventvector where ek ∈ {1, 0,−1} for 1 ≤ k ≤ K correspondsto type-k new call arrival, ek = 1, type-k call departure,ek = −1, or no change, ek = 0; and ek ∈ {1, 0} forK + 1 ≤ k ≤ 2K corresponds to type-k handoff arrival,ek = 1, or no change, ek = 0. At each decision epoch thecontroller observes the system state and, based on the eventtype, selects an action. Let Ax,e be a finite set of the availableactions in state x when event e occurs and given by

Ax,e =

⎧⎪⎨⎪⎩{1, 0}, if ek = 1 and x+

k ∈ S

{1}, if ek = 1 and x+k /∈ S

{0}, ek = −1(29)

where a = 1 means reject and a = 0 means otherwise (acceptor no-action).

Given the current state x, the current event e and the selectedaction a; then the next state y is deterministically given byy = f(x, e, a) where:

y =

⎧⎪⎨⎪⎩

x, if ek = 1 and a = 1x+k , if ek = 1 and a = 0

x−k , ek = −1(30)

The time from one decision epoch to the next decisionepoch is a continuous random variable with a conditionalprobability distribution, Fτ (t|x), given the current state ximmediately after the decision. For the above settings, i.e.,exponential inter-arrival and service times, Fτ (t|x) is alsoexponential with expected time until the next decision epoch(next event) given as

τ(x) =

[K∑k=1

xkμk +2K∑k=1

λk

]−1

(31)

The probability of the next event being an arrival ordeparture of type-k, given the current state x, is expressedas

p(x,e) =

{λkτ(x), if ek = 1 and 1 ≤ k ≤ 2Kxkμkτ(x), if ek = −1 and 1 ≤ k ≤ K

(32)

When a request of type-k is rejected, the immediate (stage)cost, c(x, e, a), incurred is⎧⎪⎨⎪⎩

wNk, if ((ek = 1 and a = 1) and 1 ≤ k ≤ K)

wH(k−K), if ((ek = 1 and a = 0) and K + 1 ≤ k ≤ 2K)

0, elsewhere(33)

A deterministic stationary allocation policy is a mapping fromstates and events to actions, i.e., π : SxE → A where E is afinite set of all events and A is a finite set of all actions.

The optimality equations for the average cost semi-Markovdecision process will have the following recursive form:

h∗(x) + g∗τ(x) = Ee

{mina∈Ai

{c(x,e,a) + h∗ (f (x, e, a))}},

∀x ∈ S(34)

EL-ALFY et al.: A LEARNING APPROACH FOR PRIORITIZED HANDOFF CHANNEL ALLOCATION IN MOBILE MULTIMEDIA NETWORKS 1657

13

2 G C-10 ….1 ….

)0()0( 1111 HHNN

1 1G 1C

)()( 1111 GG HHNN )1()1( 1111 CC HHNN

C

1)1(C2 1

Fig. 4. Generic allocation strategy: transition rate diagram for a single trafficclass.

where h∗(x) is an optimal state dependent value functionh∗ : S → R. Solving these equations results in the optimalvalues h∗(x), ∀x ∈ S and g∗. Knowing h∗(x), ∀x ∈ S, thecorresponding optimal policy is determined by

π∗(x,e) = arg mina∈Ax,e

{c(x,e,a) + h∗ (f (x, e, a))} ,∀x ∈ S and e ∈ E

(35)

We use a temporal difference approach TD(0) [24] to learnan optimal policy. The relative value functions are estimatedon-line from the generated state-action sample sequence. Ini-tially the controller parameters, value functions and policygain, are set to some values, e.g., zeros. At each decisionepoch the system state is observed and the state value functionis updated according to

hnew(x) = (1 − α)hold(x) + α{c(x,e,a) − gτ(x)+ hold (f (x, e, a))}, (36)

where again α ∈ [0, 1] is the learning rate.A number of variations are possible for the above basic

problem formulation approach by using different state def-initions, e.g., expanded states, and/or changing the decisionepochs to correspond to all events or arrivals only.

State Expansion: We expand the state vector to incorporatethe event vector as part of the state definition, i.e., s = (n, e).This change results in a dramatic reduction in the action spacecardinality although it increases cardinality of the state space.When the requested band is available, two actions are possiblein each state to admit or reject the call request. This is usefulreduction if we learn use Q-learning to learn the action valuefunctions instead of the state value functions.

Decision Epochs: Changing the decision epochs to corre-spond to the arrivals only, where actual decisions are required,results in reduction in the cardinality of the event space. Inthis case the system state may change several times betweendecision epochs due to call departures and therefore the nextstate is stochastically determined by the state transition prob-ability. This change results in complications in the dynamicprogramming algorithms since they need a prior knowledgeof a perfect transition and cost model. However, it helps thelearning schemes since they are model-free and there is noneed to learn or update the value functions at the departureevents.

2) The Single Traffic Class Case: For a single traffic class,the transition rate diagram is shown in Fig. 4 for a genericallocation policy. The optimal allocation policy that prioritizeshandoffs has a simple structure of guard threshold type.

In [12], it is proved that a guard channel approach is optimalfor a single traffic class, therefore βH1(i) = 1, for 0 ≤ i ≤ C,and βN1(i) = 1, for 0 ≤ i ≤ G and zero otherwise. G is anoptimal threshold value.

50,0

0,0

1,0

2,0

0,1 0,2 0,25

1,1 1,2 1,24

2,24

0,24

2,22,1

49,0

48,148,0

2 2 2

1

1

1

2 2 2 25 21

2 1

50 1

…

…

Fig. 5. Complete sharing: Transition rate diagram for two traffic classes withdifferent characteristics.

0 1 2 3 4 5 6

x 1 04

0

0 . 0 5

0 . 1

0 . 1 5

0 . 2

0 . 2 5

0 . 3

0 . 3 5

0 . 4

0 . 4 5

d e c is io n e p o c h s

bloc

king

pro

babi

litie

s

n e w 1 n e w 2 h a n d 1h a n d 2

C = 5 0

= [2 0 1 0 1 0 5 ]

= [1 1 ]

B = [1 2 ]

C C = [1 2 5 4 ]

Type-2

Type-1

Type-2

Type-1

Fig. 6. Simulation results of the learning algorithm for two-class example:new call and handoff blocking probabilities for each traffic type.

RL learning can be used to determine an asymptoticallyoptimal allocation policy on-line. Although we still can use theabove proposed scheme for multiple traffic classes when K =1, in this subsection we propose another scheme for which thecontroller only interacts with network when a new call arrives.Following the results of [12], it is required only to search fora control policy for new call arrivals. Therefore, the allocationcontroller always accepts handoff calls if the requested bandis available. However, it uses reinforcement learning to learnan admission policy for new call arrivals. The system state isdefined as the number of calls in progress (aggregated state)which result in reducing the state space cardinality to be |C|where C is the total number of channels available in thecell. The decision epochs correspond to new call arrivals. Thesystem state may change several times between two decisionepochs as a result of handoff arrivals or call departures whichare considered as external disturbance. The system state spaceis a finite set S = {0, 1, 2, . . . , C}. The action set available ineach state is a finite set As = {1 = reject, 0 = admit} fors ∈ {0, 1, 2, ..., C − 1} and As = {0 = reject} for s ∈ {C}.A deterministic stationary policy is a mapping from statesto actions π : S → A. Since the state transitions are nowrandom, we use the average adjusted Q-learning algorithm forcontinuous time SMDP to learn an optimal admission policyfor new call arrivals. When a new call arrives, the system state

1658 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 5, NO. 7, JULY 2006

is observed and a decision is made based the estimated actionvalue functions.

IV. SIMULATION AND NUMERICAL RESULTS

In this section the performance of the learning algorithmwill be demonstrated and compared with complete sharingand optimal guard channel reservation. We use a discreteevent simulator to generate the traffic streams for new calland handoff call requests according to mutually independentPoisson processes. The channel holding times are exponen-tially distributed.

A. Multiple Traffic Class Case

Consider a particular cell within a cellular network assigneda fixed set of channels C = [50]. There are two typesof traffic classes, K = 2, each type has two differentiatedarrival streams: new calls and handoff requests. New calls andhandoffs of type-k arrive according to mutually independentPoisson processes. The arrival rate matrix is

Λ =[λN1 λN2 λH1 λH2

]=[

20 10 10 5]

(37)Each call of type-k, whether a new call or handoff, requests

a number of channels given by bk and holds these channelsfor an exponentially distributed time with mean 1/μk where

T =[μ−1

1 μ−12

]=[

1 1]

(38)

and,b =

[b1 b2

]=[

1 2]

(39)

For complete sharing policy, there is no difference betweennew calls and handoffs. Therefore, the arrival rate vector isgiven by

Λ =[λ1 λ2

]=[

30 15]

(40)

where λk = λNk + λHk, ∀k. Let n = (n1, n2) be theaggregated state of the system, where nk is the number ofactive calls of type-k in the cell. The system state space is afinite set given by

S =

{(n1, n2) |

2∑k=1

nkbk ≤ C

}(41)

The transition rate diagram for a two-dimensional Markovchain is shown in Fig. 5. The complete sharing admissiondecision rule for type-k traffic is,

a =

{0, if n+

k ∈ S

1, elsewhere(42)

where a = 0 means accept and a = 1 means reject. Thesteady-state blocking probabilities are given by

PB1 =∑

�(n1,n2)|0≤n2≤� C

b2�,n1=�C−n2b2

b1��P (n1, n2) (43)

and

PB2 =∑

�(n1,n2)|0≤n1≤� C

b1�,n2=�C−n1b1

b2��P (n1, n2) (44)

where is the largest integer smaller than x.

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

x 104

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Decision epochs ( new call arrivals)

Blo

ckin

g pr

obab

ilitie

s

CS−NCS−HRL−NRL−HGC−NGC−H

GC−HRL−H

CS−N

CS−H

GC−N

RL−N

Fig. 7. Simulation results for comparing the blocking probabilities of thethree policies for same traffic scenario over a single run.

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

x 104

0

5

10

15

Decision epochs

Ave

rage

cos

t inc

urre

d pe

r uni

t tim

e

CSRLGC

CS

RL

GC

Fig. 8. Long-term average cost incurred per unit time over a single run withsame traffic scenario as Fig. 7.

For the given numerical values above, the analytical valuesfor the blocking probabilities are PB1 = 0.1622 and PB2 =0.3063. The allocation policy for this typical example isobtained through the learning algorithm and its simulationperformance is depicted in Fig. 6. The controller observesthe system state at each event and learns the aggregated statevalue functions on-line. When a call arrival of type-k occurs,the controller selects an action based on the estimated average-adjusted states values.

B. Single Traffic Class Case

We consider a particular cell within a cellular network withC = 30 channels, λN = 20, λH = 10, μC = 1, wN = 1 andwH = 5. The analytical solution reveals the optimal thresholdG = 28 and the corresponding blocking probabilities PHB =0.0185 and PNB = 0.2354.

The simulation-based performance for the three policies(CS: complete sharing, GC: guard channel, and RL: rein-forcement learning) is depicted in Figs. 7-12. Fig. 7 showsa single run accumulated blocking probabilities. For optimal

EL-ALFY et al.: A LEARNING APPROACH FOR PRIORITIZED HANDOFF CHANNEL ALLOCATION IN MOBILE MULTIMEDIA NETWORKS 1659

0 0.5 1 1.5 2 2.5 3

x 104

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Decision epochs

Blo

ckin

g pr

obab

ilitie

s

New Call

Handoff

Fig. 9. Simulation results for guard channel policy over 5 runs for differenttraffic scenarios.

0 0.5 1 1.5 2 2.5 3

x 104

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Decision epochs

Blo

ckin

g pr

obab

ilitie

s

New Call

Handoff

Fig. 10. Simulation results for reinforcement learning policy over 5 runs fordifferent traffic scenarios.

guard threshold, the blocking probabilities are approximatelyPHB = 0.0147 and PNB = 0.2305; and for the learningapproach PHB = 0.0248 and PNB = 0.2325 which are veryclose to the analytical solutions. Fig. 8 compares the long-runaverage cost incurred per time step. Again the performanceof the learning approach is very close to the optimal guardthreshold. Figs. 9 and 10 depict the blocking probabilities forthe guard channel and learning policies for five simulationruns. Figs. 11 and 12 show the corresponding average overthe five runs. As seen the learning approach has a comparativeperformance to the optimal guard threshold.

V. CONCLUSIONS AND FUTURE WORK

The call admission control in cellular mobile networks withprioritized handoffs has been formulated as an average costcontinuous-time Markov decision process and a reinforce-ment learning approach for finding a near-optimal admissionpolicy has been proposed. A key finding of this study isthat the reinforcement leaning algorithm has a comparativeperformance, for a single traffic class case, to the optimal

0 0.5 1 1.5 2 2.5 3

x 104

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Decision epochs

Blo

ckin

g pr

obab

ilitie

s

RL−NGC−N

Fig. 11. Simulation results for guard channel policy and learning: new callaverage blocking probabilities after 5 runs for different scenarios of Figures9 and 10.

0 0.5 1 1.5 2 2.5 3

x 104

0

0.01

0.02

0.03

0.04

Decision epochs

Blo

ckin

g pr

obab

ilitie

s

GC−HRL−H

Fig. 12. Simulation results for comparing handoff average droppingprobabilities for learning and optimal guard policies – the average is oversame 5 runs as Fig. 11.

guard threshold policy. For multiple traffic classes the learningalgorithm generalizes the concept of guard threshold to guardstates. This paper is part of a major study in which we arecurrently exploring the application of the learning algorithmfor multimedia cellular mobile networks under diverse QoSconstraints. Open research topics include the study of thehandoff-prioritized channel allocation among multiple trafficclasses in multi-cell systems. Others are to incorporate theinformation gained about the system model over time in thelearning process, and to study the performance of the learningalgorithm for non-stationary conditions.

ACKNOWLEDGMENT

The first author would like to thank King Fahd Universityof Petroleum and Minerals (KFUPM) for their support duringthis work.

REFERENCES

[1] G. L. Stüber, Principles of Mobile Communication. Norwell, MA:Kluwer Academic Publishers, 1996.

1660 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 5, NO. 7, JULY 2006

[2] S. Tekinay and B. Jabbari, “Handover and channel assignment in mobilecellular networks,” IEEE Commun. Mag., vol. 29, no. 4, pp. 42-46, Apr.1991.

[3] R. Sutton and A. Barto, Reinforcement Learning: An Introduction.Cambridge, MA: MIT Press, 1998.

[4] D. P. Bertsekas and J. N. Tsitsiklis, Neurodynamic Programming.Belmont, MA: Athena Scientific, 1996.

[5] S. P. Singh and D. P. Bertsekas, “Reinforcement learning for dynamicchannel allocation in cellular telephone systems,” in Advances in NIPS,pp. 974-980 (Cambridge, MA: MIT Press, 1997).

[6] J. Nie and S. Haykin, “A Q-learning based dynamic channel assign-ment technique for mobile communication systems,” IEEE Trans. Veh.Technol., vol. 48, no. 5, pp. 1676-1687, Sep. 1999.

[7] S.-H. Oh and D.-W. Tcha, “Prioritized channel assignment in a cellularradio networks,” IEEE Trans. Commun., vol. 40, no. 7, pp. 1259-1269,July 1997.

[8] R. Guerin, “Queueing-blocking system with two arrival streams andguard channels,” IEEE Trans. Commun., vol. 36, no. 2, pp. 153-163,Feb. 1988.

[9] D. Hong and S. S. Rappaport, “Traffic model and performance analysisfor cellular mobile radio telephone systems with prioritized and nonpri-oritized handoff procedures,” IEEE Trans. Veh. Technol., vol. 35, pp.77-92, July 1986.

[10] S. Boumerdassi and A. Beylot, “Adaptive channel allocation for wirelessPCN,” Mobile Networks and Applications, vol. 4, no. 2, pp. 111-116,June 1999.

[11] O. Yu and V. Leung, “Adaptive resource allocation for prioritized calladmission over an ATM-based wireless PCN,” IEEE J. Select. AreasCommun., vol. 15, no. 7, pp. 1208-1225, Sep. 1997.

[12] D. T. R. Ramjee and R. Nagarajan, “On optimal call admission controlin cellular networks,” Wireless Networks, vol. 3, no. 1, pp. 29-41, Jan.1997.

[13] M. Saquib and B. Yates, “Optimal call admission to a mobile cellularnetwork,” in Proc. IEEE Veh. Technol. Conf., July 1995, pp. 190-194.

[14] C. J. Ho and C. T. Lea, “Improving call admission policies in wirelessnetworks,” Wireless Networks, vol. 5, no. 4, pp. 257-265, July 1999.

[15] M. L. Putterman, Markov Decision Processes: Discrete StochasticDynamic Programming. New York: Wiley, 1994.

[16] H. C. Tijms, Stochastic Models: An Algorithmic Approach. New York:Wiley, 1994.

[17] D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont,MA: Athena Scientific, 1995.

[18] J. S. Kaufman, “Blocking in a shared resource environment,” IEEETrans. Commun., vol. 29, no. 10, pp. 1474-1481, Oct. 1981.

[19] R. B. Cooper, Introduction to Queueing Theory. New York: ElsevierScience Publishing Co. Inc., 1981.

[20] K. W. Ross and D. H. Tsang, “Optimal circuit access policies in an ISDNenvironment: A Markov decision approach,” IEEE Trans. Commun.,vol. 37, no. 9, pp. 934-939, Sep. 1989.

[21] S. Mahadevan, “Average reward reinforcement learning: Foundations,algorithms and empirical results,” Machine Learning, vol. 22, no. 1, pp.159-196, Mar. 1996.

[22] R. Guerin, H. Ahmadi, and M. Naghshineh, “Equivalent capacity andits application to bandwidth allocation in high-speed networks,” IEEEJ. Select. Areas Commun., vol. 9, no. 7, pp. 968-981, Sep. 1991.

[23] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning,vol. 8, no. 3, pp. 279-292, May 1992.

[24] R. S. Sutton, “Learning to predict by the method of temporal differ-ences,” Machine Learning, vol. 3, no. 1, pp. 9-44, Aug. 1988.

El-Sayed El-Alfy (S’00-M’01) received his M.S.degree in Computer Science and Ph.D. in Com-puter Engineering, both from Stevens Institute ofTechnology, Hoboken, NJ, in 2002. He is currentlyworking as an Assistant Professor in the ComputerScience Department, King Fahd University of Pe-troleum and Minerals, Saudi Arabia. He is teachingcourses on networking, network design, client/serverprogramming, and multimedia data compression. Heis serving in a number of standing departmentaland college committees. From 2002-2004, he was

appointed as an Assistant Professor at Tanta University, Egypt; and also asAdjunct Assistant Professor at Delta Academy of Computers, Egypt and aConsultant for ITC, Canadian Project in Egypt. From 2001-2002, he waswith Lucent Technologies Inc. as a Member of Technical Staff and was apost-doctoral fellow at Stevens Institute of Technology. From 1998-2001, hewas an Affiliated Instructor at Stevens. From 1992-1997, he worked as aGraduate Assistant in the Department of Computer and Control Engineering,Tanta University, Egypt. His current research interests include mobile anddistributed computing systems: performance analysis, design, optimization,management and applications; and applications of soft computing in network-related problems.

Yu-Dong Yao (S’88-M’88-SM’94) received theB.Eng. and M.Eng. degrees from Nanjing Universityof Posts and Telecommunications, Nanjing, China,in 1982 and 1985, respectively, and the Ph.D. degreefrom Southeast University, Nanjing, China, in 1988,all in electrical engineering.

From 1989 and 1990, he was at Carleton Univer-sity, Ottawa, Canada, as a Research Associate work-ing on mobile radio communications. From 1990to 1994, he was with Spar Aerospace Ltd., Mon-treal, Canada, where he was involved in research

on satellite communications. From 1994 to 2000, he was with QualcommInc., San Diego, CA, where he participated in research and development inwireless code-division multiple-access (CDMA) systems. He joined StevensInstitute of Technology, Hoboken, NJ, in 2000, where he is an AssociateProfessor in the Department of Electrical and Computer Engineering anda Director of the Wireless Information Systems Engineering Laboratory(WISELAB). He holds one Chinese patent and ten U.S. patents. He wasa Guest Editor for a special issue on wireless networks for the InternationalJournal of Communication Systems. His research interests include wirelesscommunications and networks, spread spectrum and CDMA, antenna arraysand beamforming, SDR, and digital signal processing for wireless systems.

Dr. Yao is an Associate Editor of IEEE COMMUNICATIONS LETTERS

and IEEE Transactions on Vehicular Technology, and an Editor for IEEETransactions on Wireless Communications.

Harry Heffes (M’66-SM’82-F’90) received hisB.E.E. degree from the City College of New York,New York, N.Y., in 1962, and the M.E.E. degreeand the Ph.D. degree in Electrical Engineering fromNew York University, Bronx, N.Y., in 1964 and 1968respectively. He joined the Department of ElectricalEngineering and Computer Science at Stevens Insti-tute of Technology in 1990. He assumed the facultyposition of Professor of Electrical Engineering andComputer Science after a twenty eight year career atAT&T Bell Laboratories. In his early years at Bell

Labs he worked on the Apollo Lunar Landing Program, where he appliedmodern control and estimation theory to problems relating to guidance,navigation, tracking and trajectory optimization. More recently his primaryconcern has been with the modeling, analysis, and overload control oftelecommunication systems and services. He is the author of over twentyfive papers in a broad range of areas including voice and data communicationnetworks, overload control for distributed switching systems, queuing andteletraffic theory and applications, as well as computer performance modelingand analysis.

Dr. Heffes received the Bell Labs Distinguished Technical Staff Award in1983 and, for his work on modeling packetized voice traffic, he was awardedthe IEEE Communication Society’s S.O. Rice Prize in 1986. He is a fellowthe IEEE for his contributions to teletraffic theory and applications. He hasserved as a United States Delegate to the International Teletraffic Congressand as an Associate Editor for Networks: An International Journal. He isa Member of Eta Kappa Nu, Tau Beta Pi, American Men and Women ofScience and is listed in Who’s Who in America.