Distributed Scheduling and Delay-Aware Routing in Multi ...yucheng/YCheng_TVT16_2.pdfSC networks into tuple-based MR-MC networks. By analyzing the drift of well-deﬁned Lyapunov functions,

0018-9545 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/TVT.2015.2474695, IEEE Transactions on Vehicular Technology

1

Distributed Scheduling and Delay-Aware Routing inMulti-Hop MR-MC Wireless Networks

Xianghui Cao, Member, IEEE, Lu Liu, Student Member, IEEE, Wenlong Shen, Student Member, IEEE,and Yu Cheng, Senior Member, IEEE

Abstract—In multi-radio multi-channel (MR-MC) networkswith significantly expanded network resource space, many ex-isting scheduling/routing algorithms rely on link based networkmodel and apply different heuristics in algorithm design in orderto achieve/approximate throughput optimality. In this paper,using a tuple-based multi-dimensional conflict graph model, weestablish a cross-layer framework which facilitates systematicallystudying distributed scheduling and routing in multi-hop multi-path MR-MC networks. In this framework, each tuple-linkis installed with a routing controller which feeds controlledamounts of data to the tuple-link output queues for schedulingand transmission. We rigorously prove that, under a set ofcertain conditions, the network is queue stable in mean senseunder the distributed maximal scheduling policy. Based onLyapunov optimization, we further propose a distributed delay-aware multi-path routing method which aims at minimizing theend-to-end delay of each commodity flow. Extensive simulationresults demonstrate that the proposed joint scheduling/routingalgorithm outperforms existing link based single-path and multi-path algorithms and tuple-based cross-layer control algorithm.

Index Terms—Multi-radio multi-channel networks, distributedscheduling, distributed delay-aware routing, queue stability

I. INTRODUCTION

Exploring the radio and channel diversities, multi-radiomulti-channel (MR-MC) wireless networks can achieve sig-nificantly larger capacities than traditional single-radio single-channel (SR-SC) networks [1]–[4]. The MR-MC networkmodel is a fundamental generalization of future wirelessnetworks such as cognitive radio networks, IEEE 802.16 basedmesh networks and the Long Term Evolution based cellularnetworks [5]–[7].

The problem of throughput-optimal scheduling of funda-mental importance for achieving optimal performance whilemaintaining network stability has attracted much attentionrecently. In SR-SC networks, there are many works apply-ing the back-pressure based algorithm to schedule links fordata transmission [8], [9], where the scheduling decisionsare usually made in centralized manners. The back-pressure

Copyright (c) 2015 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposes must beobtained from the IEEE by sending a request to [email protected].

This work was supported in part by the NSF under Grant CNS-1320736and CAREER Award Grant CNS-1053777, and National Natural ScienceFoundation of China under grant 61203036.

X. Cao is with the School of Automation, Southeast University, Nanjing,China. (Email: [email protected]).

L. Liu, W. Shen and Y. Cheng are with the Department of Electricaland Computer Engineering, Illinois Institute of Technology, Chicago, USA.(Email: lliu41,[email protected], [email protected]).

scheduling is equivalent to solving a maximum weightedindependent set (MWIS) problem, which is proven to be NP-hard. Suboptimal scheduling algorithms have been proposedto approximate throughput optimality with low computationcomplexity [10]–[12]. One popular example is the greedy max-imal scheduling (also known as the longest queue first policy)which is based on the greedy MWIS algorithm [10]. Sincecentralized scheduling policies are not favored in large-scalenetworks, distributed scheduling policies have been studied.For example, in [13], distributed randomized scheduling poli-cies are proposed where either each link randomly decides toparticipate in scheduling or the nodes randomly decide to setupconnections between each other. As another approach, thedistributed maximal scheduling allows at least one backloggedlink in the interference set of a link be scheduled [14]. It hasbeen shown that such an algorithm can achieve a capacityefficiency ratio (the largest achievable fraction of the optimalcapacity region while guaranteeing network stability) of 1

Kwhere K is the interference degree, i.e., the maximum numberof non-interfering links in the interference set of any link inthe network [15].

In MR-MC networks, with significantly increased networkdimension, link scheduling is often coupled with radio/channelassignment. There are only a few studies on designing andanalyzing the performance of distributed scheduling policies inMR-MC networks. In [16], the distributed maximal schedulingpolicy has been applied in MR-MC networks based on link-channel pairs (LCPs), where a physical link is split into severalLCPs with each LCP maintaining a queue to be consideredin the maximal scheduling. A systematic approach that trans-forms an MR-MC network into an equivalent virtual SR-SC network through a tuple-based multi-dimensional conflictgraph (MDCG) has been proposed in [17]. However, theanalysis is constrained within single-hop networks.

Despite its advantage in achieving throughput-optimalscheduling in single-hop networks, the back-pressure basedalgorithm in multi-hop scenarios may incur excessively longdelays and may not guarantee stability of the entire network[18]. On top of the scheduling policy, the network layerrouting algorithm also plays a vital role for achieving optimalperformance in multi-hop scenarios. There are several studieson joint scheduling and routing design [19], [20]. The back-pressure based scheme in [19] minimizes the lengths of thepaths that can guarantee network stability. In [21], a joint con-gestion control, routing and scheduling algorithm is proposedwhich can guarantee both requirements of data rate and end-to-end delay. Several routing metrics for minimizing end-to-



2

end delay in MR-MC networks are designed and implementedin [22]. However, many of them rely on centralized routing,which may be impractical for large-scale multi-hop networks.A distributed multi-path routing policy has been proposed in[16] in which the source node of a flow determines the fractionof the flow to be routed on each of the pre-determined paths.The fraction decisions are made by solving an optimizationproblem which is based on the heuristic of minimizing thecongestion cost (which can be also viewed as total queueingdelay). A distributed path selection algorithm is designed in[17], where the optimal path of a flow is selected such that theweighted sum of queueing delay and number of hops alongthe path is minimized. Since end-to-end delay consists of notonly queueing delay but also transmission delay as well asthe delay due to the scheduling policy used, neither of theabove routing methods manifests the true delay in the routingdecision process.

In MR-MC networks with a multi-dimensional resourcespace, many existing scheduling/routing algorithms rely onlink based network model and apply different heuristics inalgorithm design in order to achieve/approximate throughputoptimality. Different from these studies, in this paper, wepropose a cross-layer framework which facilitates systematicstudy of distributed scheduling and routing in multi-hop multi-path MR-MC networks. The framework is established basedon the tuple-link based MDCG network model where a tuple isdefined as a node-radio-channel resource vector in the multi-dimensional resource space. The transmitter tuple of eachtuple-link implements a routing controller which feeds datatraffic to the link’s output queue for further scheduling andtransmission. An important role of the controller is to shapethe incoming traffic from preceding tuple-links along the pathsof a flow so as to maintain network stability. The function ofthe controller is similar to that of the regulator proposed in[15], [17]; a major difference in our case is that the parameterstaken by the controllers are important routing variables thatsignificantly vary the end-to-end delay performance.

We study the distributed joint scheduling and routing in thecontext of multi-commodity flow problem. We assume a set ofpre-discovered potential paths for each flow and that each pathdelivers a fraction, which can be tuned dynamically, of the totalamount of arrivals of the corresponding flow. The schedulingalgorithm extends the maximal scheduling in traditional SR-SC networks into tuple-based MR-MC networks. By analyzingthe drift of well-defined Lyapunov functions, we show that,under certain conditions, the network queue stability underdistributed maximal scheduling is guaranteed in mean sense.Furthermore, based on the Lyapunov optimization method,we propose a multi-path routing optimization problem whichaims at minimizing the end-to-end delay of each flow withboth queueing delay and scheduling delay taken into account.Based on this, we develop a distributed delay-aware routing(DDAR) algorithm to decide the traffic fractions taken bythe paths, where we apply the minimum-consensus algorithm[23] to ensure that the constraints of the routing optimizationproblem can be satisfied in a distributed manner. Finally, weconduct extensive simulations to evaluate the performance ofthe proposed method. The results show that our method outper-

forms existing algorithms including the link based single-pathand multi-path schemes proposed in [16] and the cross-layercontrol based scheme proposed in [17].

The remainder of this paper is organized as follows. SectionII presents system model. The scheduling policy and routingalgorithm are described in Section III and IV, respectively.Section V presents simulation results, and Section VI con-cludes this paper.

II. SYSTEM MODEL

In this section, we show that an MR-MC network can berepresented as a tuple-based virtual SR-SC network, basedon which we propose a cross-layer framework for designingscheduling and routing algorithms. We also describe the queuedynamics in such a framework. Notations used throughout thispaper are summarized in Table I.

A. Tuple-based Generic Model of MR-MC Networks

Consider an MR-MC wireless network as an undirectedgraph Gp(N ,Lp) with node set N and physical link set Lp,where we assume each node uses the same transmit power.Each node i is equipped with a set of radios and each of theradios can operate on a set of orthogonal channels. A tuplepi = (ni, ri, c) is a vector in the multi-dimensional networkresource space, which means the resource allocation of nodeni, one of its radios ri and its operating channel c. A tuple-link ℓ = (pi, pj) = ((ni, ri, c), (nj , rj , c)) represents that thetransmission from the radio ri of node ni to the radio rj ofnode nj is carried out over channel c. For any tuple-link ℓ, itscapacity wℓ is also the capacity of ℓ’s corresponding physicallink. Based on the tuple-link model, a physical link connectingnodes ni and nj can be mapped into |Mi|×|Mj |×|C| tuple-links with each link indicating an assignment of radios andchannel. With the fine-grained tuple-based modeling technique[5], the whole network can be mapped into a virtual SR-SCnetwork as graph G(T ,L) with vertex (tuple) set T and tuple-link set L. In the following, if not specified, the terms link andtuple-link are used interchangeably.

Two tuple-links ℓ1 and ℓ2 are said conflicting with eachother (i.e., they cannot be simultaneously active for data trans-mission) if either of the following conditions is met: (1) theirbeginning or ending tuples share a common radio; (2) theyoperate on the same channel and are physically within eachother’s interference range. Based on the conflict relationshipsamong all the tuple-links, a multi-dimensional conflict graph(MDCG) Gc(L,X ) can be established with vertices as all thetuple-links and edges as the set of conflict relationships. Foreach ℓ ∈ L, denote Iℓ as the set of all tuple-links that haveedges with ℓ in X . By convention, let ℓ ∈ Iℓ. Assumingthe protocol interference model [17] with which a schedulingpolicy should make sure that, if ℓ is scheduled, none of thelinks in Iℓ other than ℓ is scheduled simultaneously. For ℓ,define its interference degree Kℓ as the maximum number oftuple-links in Iℓ that can be scheduled at the same time. LetK = maxℓ Kℓ be the network interference degree.

We study the scheduling and routing issues in the contextof multi-commodity flow problem in which there are a set of



3

TABLE INOTATIONS

Notation MeaningN Set of all nodes in the networkMi Set of all radios of node iC Set of orthogonal channels available to the network

Lp, L Sets of physical links and tuple-links, respectivelyF Set of all flowsλf Average packet arrival rate at the source of flow f

b(ℓ), e(ℓ) Beginning and ending nodes of link ℓ, respectivelyc(ℓ), wℓ Operating channel and capacity of link ℓ, respectivelys(f), d(f) Source and destination nodes of flow f , respectively

K Network interference degreePf Set of candidate paths for flow fPf,i Stands for path i of flow f , where i ∈ Pf

Lf,i Set of links on the path Pf,i. Lf = Lf,i|i ∈ Pfηf,i The probability that the incoming packets of flow f will

be routed through path Pf,i

hf,iℓ Hop count, i.e., number of links preceding ℓ along the

path Pf,i

Bf,iℓ ,Nf,i

ℓ The links immediately precede and follow ℓ along thepath Pf,i, respectively

Qf,iin,ℓ,Qf,i

out,ℓ Input and output queues maintained by the transmitternode of link ℓ for flow f over path Pf,i, respectively,where ℓ ∈ Lf,i

Df,iℓ Amount of packets delivered from Qf,i

in,ℓ to Qf,iout,ℓ

T f,iℓ Amount of packets actually transmitted from Qf,i

out,ℓ to

Qf,i

in,Nf,iℓ

over link ℓ

µf,i

hf,iℓ

Average output rate of the routing controller of the h-thlink on path Pf,i, if Qf,i

in,ℓ(t) > Γwℓ, where Γ ≥ 1

flows F injected into the network. For each flow f ∈ F , theaverage traffic arrival rate at its source node is λf . Take thesimple network shown in Fig. 1(a) for example, where eachnode has one or two radios and there are one channel available(thus each node has one or two tuples as shown in this figure).There are two commodity flows between two pairs of sourceand destination nodes, respectively. Each flow may go throughmultiple multi-hop paths until reaching its destination. Theproblem is to design proper scheduling and routing policies todeliver as much flow to corresponding destinations as possiblewith low end-to-end delay. We consider a slotted system inwhich time is divided into slots of unit length. The schedulingand routing decisions are made at the beginning of each slot.

B. Queue Dynamics

We assume that each commodity flow f is associatedwith a set of candidate paths Pf . The incoming packets offlow f in slot t will be routed through the path Pf,i withprobability ηf,i(t). By applying the MDCG based off-lineplanning technique [17], a relatively small set of Pf and thecorresponding long-term average of ηf,i for each flow f canbe obtained in a centralized manner. Alternatively, as discussedin [16], the candidate set Pf can be discovered online by thesource node of each flow. In the following, we assume thatthe paths Pf have been discovered and are fixed over time.

The transmitter node of link ℓ, i.e., b(ℓ), maintains both aninput and an output queues, i.e., Qf,i

in,ℓ and Qf,iout,ℓ, respectively,

to buffer the packets of flow f to be delivered over path Pf,i,where ℓ is on path Pf,i. For each link ℓ, its transmitter nodeb(ℓ) is installed with a routing controller which maintains the

1s

2s

2d

1d

(a) Multi-commodity flow problem. Each circle means a tuple

Routing

controller

Routing

controller

Routing

controller

(b) Input and output queues

Fig. 1. System models.

set of input queues Qf,iin,ℓ|f ∈ F , i ∈ Pf , ℓ ∈ Lf,i and

feeds controlled amounts of data to the corresponding outputqueues for transmission. If ℓ is on none of the candidate pathsof flow f , we simply have Qf,i

in,ℓ(t) ≡ 0 and Qf,iout,ℓ(t) ≡ 0. In

the following, for ease of exposition, we focus only on linkssuch that each of them is on at least one path. That is, byusing Qf,i

in,ℓ and Qf,iout,ℓ, we imply that ℓ ∈ Lf,i. The input and

output queues are illustrated in Fig. 1(b).

The routing controller of link ℓ is characterized by a setof parameters µf,i

ℓ |ℓ ∈ Lf,i. As shown later in this paper,the routing controller plays an important part in both ensuringnetwork stability and making routing decisions.

1) Dynamics of input queue Qf,iin,ℓ: If b(ℓ) is the source of

flow f , for each path Pf , the arrivals of the flow will bebuffered in Qf,i

in,ℓ with probability ηf,i; otherwise, if b(ℓ) isnot the source of flow f , Qf,i

in,ℓ buffers packets from Bf,iℓ . The

routing controller of ℓ randomly checks Qf,iin,ℓ with probability

1Γ , where Γ ≥ 1. At the beginning of the check-point slot(say, slot t), if Qf,i

in,ℓ(t) > Γwℓ, Γwℓ amount of packets willbe transferred1 from the input queue to the correspondingoutput queue Qf,i

out,ℓ with probability 1wℓ

µf,i

hf,iℓ

(t), where hf,iℓ ∈

0, 1, . . . , |Lf,i|−1 is the hop count. Otherwise, nothing willbe delivered from Qf,i

in,ℓ to Qf,iout,ℓ. At slot t, for each link ℓ,

the expected amount of packets actually delivered from Qf,iin,ℓ

1Since the transferring is performed within a node, we assume that it canbe done instantly and the amount is not constrained by the link capacity wℓ.



4

to Qf,iout,ℓ is

E[Df,iℓ (t)] =

1

Γ1Qf,i

in,ℓ(t)>ΓwℓΓwℓ

µf,i

hf,iℓ

(t)

wℓ

= µf,i

hf,iℓ

(t)1Qf,iin,ℓ(t)>Γwℓ

, (1)

where 1condition is 1 if “condition” holds and is 0 if otherwise.Further, define Tℓ as the total amount of transferred data outfrom the routing controller of link ℓ, i.e,

Dℓ(t) ,∑f∈F

∑i∈Pf

Df,iℓ (t). (2)

Then, the dynamics of the input queue can be written as:

Qf,iin,ℓ(t+ 1) =

[Qf,i

in,ℓ(t)−Df,iℓ (t)

]++T f,i

Bf,iℓ

(t)1b(ℓ)=s(f)

+ λf (t)ηf,i(t)1b(ℓ)=s(f). (3)

Remark 1: The input queues can be viewed as buffers tocope with arrival fluctuations. By using Γ, the arrivals areassembled into blocks of packets which provide more stableinput to the output queues. Γ can be set to 1, which allowsthat each input queue is checked in every time slot. Othertransmission rules are also allowed as long as the averageamount of packets delivered from input to output queues isunder controlled by the decision variable µf,i

h . The conditionQf,i

in,ℓ(t) > Γwℓ may be also dropped, in which case the queuestability theorem still holds.

2) Dynamics of output queue Qf,iout,ℓ: Define the total length

of the output queues of link ℓ as

Qout,ℓ(t) ,∑f∈F

∑i∈Pf

Qf,iout,ℓ(t). (4)

A link ℓ is said backlogged if its total output queue lengthexceeds Γwℓ, i.e., Qout,ℓ(t) > Γwℓ, where Γ > 02. At eachslot, only backlogged links will be considered for scheduling.Define a binary variable πℓ as follows. If ℓ is not scheduled,πℓ(t) = 0 and nothing will be transmitted over link ℓ at slott. If ℓ is backlogged and scheduled at slot t, πℓ(t) = 1 andtotally wℓ amount of data will be transmitted out from Qout,ℓ tothe incoming queues of next-hop links. The detailed strategycan be as follows. For each pair of (f, i) such that ℓ ∈ Lf,i,Qf,i

out,ℓQout,ℓ

wℓ amount of data will be transmitted from Qf,iout,ℓ to

Qf,i

in,Nf,iℓ

. It is worth noticing that, as indicated later by theproof of Theorem 1, how wℓ is distributed to the transmissionsbetween (Qf,i

out,ℓ, Qf,i

in,Nf,iℓ

) does not affect the network queuelength stability.

At slot t, if ℓ is backlogged, the amount of packets actuallytransmitted from Qf,i

out,ℓ to Qf,i

in,Nf,iℓ

over ℓ is

T f,iℓ (t) =

Qf,iout,ℓ(t)

Qout,ℓ(t)wℓπℓ(t), (5)

Tℓ(t) ,∑f∈F

∑i∈Pf

T f,iℓ (t) = wℓπℓ(t). (6)

2For a more generic backlog criterion, it is also possible to use time-varyingΓ without affecting the stability analysis presented later in this paper (see theinequality (40)). For simplicity, we still use constant Γ in the following.

The dynamics of the output queue is

Qf,iout,ℓ(t+ 1) = Qf,i

out,ℓ(t)− T f,iℓ (t) +Df,i

ℓ (t). (7)

C. Scheduling and routing

Based on the above framework, the scheduling and routingdecisions are made by the routing controllers. Specifically, adistributed scheduling policy should be able to decide πℓ(t)based on only local information (particularly the informationshared within Iℓ). A routing algorithm should determine thevalues for µf,i

hf,iℓ

(t)|ℓ ∈ Lf,i and ηf,i(t). The schedulingand routing methods must ensure the network queue stabilityin mean sense, i.e., limT→∞

1T

∑Tt=1

∑ℓ∈L E[Qout,ℓ(t)] < ∞

and limT→∞1T

∑Tt=1

∑ℓ∈L E[Qin,ℓ(t)] < ∞. In the follow-

ing two sections, we develop a maximal scheduling baseddistributed scheduling algorithm and a delay-aware routingalgorithm, respectively.

III. DISTRIBUTED SCHEDULING ALGORITHM

In this section, we first propose a maximal-scheduling-based distributed link scheduling policy for multi-hop MR-MC networks with a generic routing strategy. We then studythe throughput and the capacity efficiency of the proposedscheduling policy. Based on the MDCG, the distributed max-imal scheduling for SR-SC networks can be readily extendedto that for MR-MC networks as follows.

Distributed maximal scheduling algorithm: starting withthe set L, we randomly pick up a link, say ℓ, and checkwhether it is backlogged or not. If so, ℓ is scheduled for datatransmission; meanwhile, all the links in Iℓ\ℓ are removedfrom L and will not be considered for scheduling. Then, werun the above process based on the residual set L\Iℓ. Thealgorithm stops when the residual set becomes empty.

1) Stability conditions: By the above maximal schedulingpolicy, if ℓ is backlogged, at least one of the links in Iℓ isscheduled, i.e., ∑

ℓ′∈Iℓ

πℓ′(t) ≥ 1. (8)

Theorem 1 (Queue stability): For a multi-hop MR-MCnetwork, if ∃ amax ∈ (0, 1) and ∃ ϵmin > 0 such that thefollowing conditions hold for all t:

∑ℓ′∈Iℓ

∑f∈F

∑i∈Pf

µf,i

hf,i

ℓ′(t)

wℓ′1Qf,i

in,ℓ′ (t)>Γwℓ′≤ amax, ∀ℓ; (9)

µf,i0 (t) ≥ λfη

f,i(t), ∀f, i; (10)

µf,i

hf,iℓ +1

(t) ≥ ϵmin, ∀f, i, ℓ such that

Qf,iin,ℓ(t) ≤ Γwℓ and Qf,i

in,Nf,iℓ

(t) > ΓwNf,iℓ

; (11)

µf,i

hf,iℓ +1

(t)− µf,i

hf,iℓ

(t) ≥ ϵmin, ∀f, i, ℓ such that

Qf,iin,ℓ(t) > Γwℓ and Qf,i

in,Nf,iℓ

(t) > ΓwNf,iℓ

, (12)

then, the network is queue-length stable under the abovedistributed maximal scheduling algorithm.

The proof is provided in Appendix A. Basically, the proofconsists of three steps. In the first step, the input queues of



5

each flow’s source node are proven stable. Then, we showthat both Qout and Qin,N are stable in mean sense, whereQout and Qin,N are the vectors constructed by all Qf,i

out,ℓand all Qf,i

in,Nf,iℓ

, respectively. This is achieved by analyzingthe drift of the following Lyapunov function:

V (Qin,N ,Qout) = V1(Qout) + ξV2(Qin,N ,Qout), (13)

where ξ > 0 and

V1(Qout) ,1

2

∑ℓ∈L

Qout,ℓ

wℓ

∑ℓ′∈Iℓ

Qout,ℓ′

wℓ′. (14)

V2(Qin,N ,Qout) ,1

2

∑ℓ∈L

∑f∈F

∑i∈Pf

Qf,iout,ℓ +Qf,i

in,Nf,iℓ

wℓ

2

.

(15)

Apparently, V (Qin,N ,Qout) is radially unbounded and strictlypositive unless Qf,i

in,Nf,iℓ

= 0 and Qf,iout,ℓ = 0 for all f, i, ℓ.

2) Scheduling efficiency: As compared to centralized op-timal scheduling algorithms, the above distributed maximalscheduling which relies only on local information is subop-timal. To evaluate its performance, we define the followingterms and performance metrics. Define the capacity region ofa scheduler as the set of input λ under which the networkremains stable, where λ = λ1, . . . , λ|F|]. The optimal ca-pacity region Λopt of the network is defined as the supremumof all schedulers. A scheduler is throughput-optimal if it canachieve the optimal capacity region. It is thus interesting toinvestigate its scheduling efficiency. For such a suboptimalscheduler, define the capacity efficiency ratio as the largestγ such that the network is stable under any λ ∈ γΛopt. Forthe proposed scheduling algorithm, its scheduling efficiency iscaptured in the following Theorem. The proof can be foundin Appendix B.

Theorem 2 (Scheduling efficiency): For a multi-hop MR-MC network, the tuple-based distributed maximal schedulingcan achieve a capacity efficiency ratio of 1

K .

IV. DISTRIBUTED ROUTING ALGORITHM

In this section, we propose a distributed routing algorithmwhere the routing decisions are made at routing controllersof the source nodes. The routing decisions are based onsolving a linear programming obtained by using the Lyapunovoptimization technique.

A. Routing based on Lyapunov Optimization

For ease of presentation, ∀f, i, ℓ, letQf,i

out,ℓ(t) = Qf,iout,ℓ(t)1Qout,ℓ(t)>Γwℓ

Qout,ℓ(t) = Qout,ℓ(t)1Qout,ℓ(t)>Γwℓ

µf,i

hf,iℓ

(t) = µf,i

hf,iℓ


.

(16)

Consider the Lyapunov function V as in (13). Combining(40), (41) and (43) in the proof of Theorem 1, we get

E[∆V (t)|Qin,N (t),Qout(t)]

≤∑ℓ∈L

Qout,ℓ(t)

wℓ

(∑ℓ′∈Iℓ

∑f ′∈F

∑i′∈Pf′

µf ′,i′

hf′,i′ℓ′

(t)

wℓ′− 1

)

+ ξ∑ℓ∈L

∑f∈F

∑i∈Pf

Qf,iout,ℓ(t) +Qf,i

in,Nf,iℓ

(t)

w2ℓ

×(µf,i

hf,iℓ

(t)− µf,i

hf,iℓ +1

(t))+ C6

=∑ℓ∈L

Qout,ℓ(t)

wℓ

∑ℓ′∈Iℓ

∑f ′∈F

∑i′∈Pf′

µf ′,i′

hf′,i′ℓ′

(t)

wℓ′

− ξ∑f∈F

∑i∈Pf

∑ℓ∈L

µf,i

hf,iℓ

(t)

(Qf,i

out,Bf,iℓ

(t) +Qf,iin,ℓ(t)

w2Bf,i

ℓ

−Qf,i

out,ℓ(t) +Qf,i

in,Nf,iℓ

(t)

w2ℓ

)−∑ℓ∈L

Qout,ℓ(t)

wℓ+ C6. (17)

where C6 is a constant. In the above, for convenience, wedefine Qf,i

out,Bf,iℓ

= 0 if b(ℓ) = s(f) and Qf,iin,ℓ = 0 if e(ℓ) =

d(f). Supposing that the MDCG is un-directional, we havethat ∀ℓ, ℓ′ ∈ L, if ℓ′ ∈ Iℓ, then ℓ ∈ Iℓ′ . Thus, we obtain (18)(which is shown in the next page), where αf,i

ℓ (t) = ω1−ξω2 isa function of Qout,ℓ′(t)ℓ′∈Iℓ

, Qf,i

out,Bf,iℓ

(t), Qf,iin,ℓ(t), Q

f,iout,ℓ(t)

and Qf,i

in,Nf,iℓ

(t). For convenience, we further define Jf,iℓ (t) =

αf,iℓ (t)1

Qf,iin,ℓ(t)>Γwℓ

wℓµf,i

hf,iℓ

(t).

Remark 2: In (18), ∀(f, i),∑

ℓ∈Lf,i Jf,iℓ (t) can be inter-

preted as follows.

• The term ω1 can be further written as

ω1 =Qout,ℓ(t)

wℓ+

∑ℓ′∈Iℓ\ℓ

Qout,ℓ′(t)

wℓ′, (19)

where the first part represents the queueing delay in theoutput queues (since the transmission rate wℓ is limitedand we assume the queues are first-in-first-out) whilethe second represents the delay due to our distributedscheduling policy.

• For each link ℓ such that ℓ ∈ Lf,i, let Qf,iℓ = Qf,i

out,ℓ +

Qf,i

in,Nf,iℓ

be the total length of both input and outputqueues associated with ℓ with respect to flow f and pathPf,i. Then, the input and output of Qf,i

ℓ are managed bythe routing controllers with parameters µf,i

hf,iℓ

and µf,i

Nf,iℓ

.

We can view the routing controller with parameter µf,i

hf,iℓ

connecting between Qf,iℓ and Qf,i

Nf,iℓ

as a virtual link ιℓ.Thus, the term ω2 in (18) can be viewed as the queuelength (or backlog) over the virtual link ιℓ. Based on(1), µf,i

hf,iℓ

can be viewed as the link rate of ιℓ. Thus,according to [17], the throughput-optimal policy by only



6

E[∆V (t)|Qin,N (t),Qout(t)]

≤∑f∈F

∑i∈Pf

∑ℓ∈Lf,i

µf,i

hf,iℓ

(t)

wℓ

∑ℓ′∈Iℓ

Qout,ℓ′(t)

wℓ′︸︷︷︸ω1

−ξ

(Qf,i

out,Bf,iℓ

(t) +Qf,iin,ℓ(t)

wBf,iℓ

wℓ

wBf,iℓ

−Qf,i

out,ℓ(t) +Qf,i

in,Nf,iℓ

(t)

wℓ

)︸︷︷︸

ω2

−∑ℓ∈L

Qout,ℓ(t)

wℓ+ C6

,∑f∈F

∑i∈Pf

∑ℓ∈Lf,i

αf,iℓ (t)1Qf,i

in,ℓ(t)>Γwℓ

wℓµf,i

hf,iℓ

(t)−∑ℓ∈L

Qout,ℓ(t)

wℓ+ C6, (18)

considering the virtual links can be expressed as:

max :∑i∈Pf

∑ℓ∈Lf,i

µf,i

hf,iℓ

(t)

wℓω2

⇔min : −∑i∈Pf

∑ℓ∈Lf,i

µf,i

hf,iℓ

(t)

wℓω2.

In this sense, −ω2 can be viewed as a measure of thevirtual link queueing delay which is caused by both therouting decisions and queuing in the input queue.

Sinceµf,i

hf,iℓ

(t)

wℓis the probability of data transferring

from the input queue to the output queue of ℓ, the term∑ℓ∈Lf,i

µf,i

hf,iℓ

(t)

wℓαf,iℓ =

∑ℓ∈Lf,i

µf,i

hf,iℓ

(t)

wℓ(ω1 − ξω2) can be

viewed as the expected end-to-end delay along the path Pf,i,which consists of the delay due to distributed scheduling,routing as well as queueing in both the input and output queuesof the links. Thus, by minimizing this value, we can improvethe overall end-to-end delay performance.

With the aim to minimize the Lyapunov drift, we obtain arouting method by solving the following optimization problem.

Problem 1 (Centralized routing optimization problem):Find the optimal µf,i

hf,iℓ

(t)|Qf,iin,ℓ > Γwℓ3 and ηf,i(t) to

min∑f∈F

∑i∈Pf

∑ℓ∈Lf,i

Jf,iℓ (t) (20)

s.t. (9)-(12) hold, (21)∑i∈Pf

ηf,i(t) = 1, ∀f, (22)

µf,i

hf,iℓ

(t) > 0, ∀f, i, ℓ,Qf,iin,ℓ > Γwℓ, (23)

ηf,i(t) ≥ 0, ∀f, i. (24)

Remark 3: In the literature, there are two representativerouting algorithms based on the distributed maximal schedul-ing for MR-MC networks. In [16], for the traffic of flowf , the multi-path routing algorithm aims at minimizing thecongestion cost which is the summation of the queue lengths

3In case Qf,iin,ℓ ≤ Γwℓ, according to our framework, nothing will be

delivered from the input queue Qf,iin,ℓ. Thus, we do not need to consider

µf,i

hf,iℓ

(t) in this case.

along the paths. Alternatively, the objective of the cross-layercontrol based routing algorithm in [17] is to minimize thetraffic incurred within the network to support all commodityflows. The algorithm is further reduced to select the optimalpath for each flow such that the weighted sum of the pathlength and the queue lengths along this path is minimized.However, these algorithms only focus on queueing delay inmaking routing decisions, while the scheduling delay inducedby the distributed scheduling policy has not been considered.

The optimal solution to Problem 1 can only be obtainedin a centralized way, which is inefficient for network withmany flows. In the following, we propose a distributed routingalgorithm to approximate the centralized optimal solution.

B. Distributed Routing Algorithm Design

For each path Pf,i, in calculating the summation of Jf,iℓ (t),

we only need to consider those links on the paths of flow f ,i.e., ℓ ∈ Lf,i. The other information needed in calculatingJf,iℓ (t) that relates to Qf,i

in,ℓ and Qf,iout,ℓ can be collected by ℓ

itself and then transmitted to the source node. Therefore, theobjective function

∑ℓ∈Lf,i J

f,iℓ (t) can be evaluated locally by

the source node of flow f .

Although the conditions (10)-(12) are fully distributed, thecondition (9) requires global information; that is, each sourcenode of a flow needs to know the decisions of other flows.In the following, we propose a distributed method to decideµf,i

hf,iℓ

(t)|Qf,iin,ℓ > Γwℓ to satisfy condition (9) sufficiently.

∀f, i and ℓ such that Qf,iin,ℓ > Γwℓ, let

ϵf,ihf,iℓ

(t) , µf,i

hf,iℓ

(t)− µf,i

hf,iℓ −1

(t)1Qf,i

in,Bf,iℓ

>ΓwB

f,iℓ

, (25)

Corresponding to (11) and (12), ϵf,ih (t) ≥ ϵmin, where ϵmin

can be chosen sufficiently small. Meanwhile, we assume thatϵf,ihf,iℓ

(t) ≤ ϵmax(t) with ϵmax(t) as a global value which will

be discussed later. Let let µf,i0 (t) = λfη

f,i(t). Thus, along thepath Pf,i, (26) and (27) as shown in the next page can be



7

µf,i

hf,iℓ


= µf,i0 (t)

∏ℓ′:hf,i

ℓ′ ≤hf,iℓ

1Qf,i

in,ℓ′>Γwℓ′+

∑ℓ′:hf,i

ℓ′ ∈[1,hf,iℓ ]

∏ℓ′′:hf,i

ℓ′′ ∈[hf,i

ℓ′ ,hf,iℓ ]

1Qf,i

in,ℓ′′>Γwℓ′′

ϵf,ihf,i

ℓ′(t)

, µf,i0 (t)Ξf,i

0,ℓ(t) + Ξf,i1,ℓ(t) (26)

≤ µf,i0 (t) + ϵmax(t)

∑ℓ′:hf,i

ℓ′ ∈[1,hf,iℓ ]

∏ℓ′′:hf,i

ℓ′′ ∈[hf,i

ℓ′ ,hf,iℓ ]

1Qf,i

in,ℓ′′>Γwℓ′′︸︷︷︸Ξf,i

2,ℓ(t)

. (27)

proven based on (25). Then, ∀ℓ ∈ L,

∑ℓ′∈Iℓ

∑f∈F

∑i∈Pf

µf,i

hf,i

ℓ′(t)

wℓ′1Qf,i

in,ℓ′ (t)>Γwℓ′

≤∑ℓ′∈Iℓ

∑f∈F

λf

wℓ′1ℓ′∈Lf + ϵmax(t)

∑ℓ′∈Iℓ

∑f∈F

∑i∈Pf

Ξf,i2,ℓ′(t)

wℓ′,

(28)

where Ξf,i2,ℓ′(t) is defined in (27). If it holds that the flow input

vector λ ∈ Λopt, the first term to the right hand side (R.H.S.)of (28) is less than 1 as shown in the proof of Theorem 2.Then, it is feasible to choose a amax < 1 and a small enoughϵmax(t) to make sure that the R.H.S. of (28) is less thanamax. Consequently, the conditions (9)-(12) can be satisfied.Thus, our distributed optimization problem can be formulatedas follows.

Problem 2 (Distributed routing optimization problem):For each flow f , find the optimal routing parametersηf,i(t)|i ∈ Pf and ϵf,i

hf,iℓ

(t)|Qf,iin,ℓ > Γwℓ to

min∑i∈Pf

∑ℓ∈Lf,i

Jf,iℓ (t) (29)

s.t. (25) holds, ∀i, ℓ, Qf,iin,ℓ > Γwℓ, (30)

R.H.S. of (28) < amax, ∀ℓ ∈ Lf , (31)

µf,i0 (t) = λfη

f,i(t), ∀i ∈ Pf , (32)∑i∈Pf

ηf,i(t) = 1, (33)

ϵf,ihf,iℓ

(t) ∈ [ϵmin, ϵmax(t)], ∀i ∈ Pf , (34)

ηf,i(t) ≥ 0, ∀i ∈ Pf . (35)

To solve this problem, the first step is to determine theglobal value ϵmax(t) in a distributed manner.

Step 1: For each flow f , let Fℓ be the set of flows whosepaths traverse link ℓ. Thus, for the source node of each flow f ,it can obtain a local estimate of ϵmax(t), denoted as ϵfmax(t),such that the R.H.S. of (28) is less than or equal to amax

for all ℓ on the paths of f . Specifically, each link ℓ collectsall available λf |ℓ ∈ Lf and Qf,i

in,ℓ(t), Qf,iout,ℓ(t)|ℓ ∈ Lf,i

and sends them to the source nodes of the flows traversingit. Thus, ϵfmax(t) can be obtained locally by the source nodeof f based on (31). Then, all the source nodes can apply

a minimum-consensus algorithm (which runs in a fully dis-tributed manner) [23] as follows to achieve a globally feasibleϵmax(t) = minf∈Fϵfmax(t).

Minimum-consensus algorithm for computing ϵmax(t): Thealgorithm runs iteratively. Initially, each source node decidesits own ϵfmax(t) based on the above discussion. In eachiteration, each source node broadcasts its current ϵfmax(t) toits immediate neighbors. Upon receiving an ϵfmax(t), a sourcenode compares the received ϵfmax(t) and that of its own anduses the smaller one of them to update its own value; whilea non-source node will simply forward the smallest ϵfmax(t)it has received to its neighbors. The algorithm will finallyconverge to the maximum of the initial values held by thesource nodes.

Step 2: With ϵmax(t) obtained in Step 1, the source node ofeach flow can make routing decisions by solving Problem 2.Once the problem is solved, the source node will calculateµf,i

hf,iℓ

(t)|hf,iℓ ≥ 1, Qf,i

in,ℓ > Γwℓ based on (27) and then

transmit them along each path Pf,i to the correspondingrouting controllers. For those links with Qf,i

in,ℓ ≤ Γwℓ, sincethe corresponding routing controller does nothing in this case,we can simply assign µf,i

hf,iℓ

(t) = 0.

Theorem 3: If at some time t there holds that αf,iℓ (t) > 0

for all f, i, ℓ, then the optimal solutions for Problem 1 andProblem 2 coincide at this time. Moreover, the optimal solutionis: ∀f ∈ F and i ∈ Pf ,

ηf,i(t) =

1, if i = argmin∑

ℓ∈Lf,i

αf,iℓ (t)

wℓΞf,i0,ℓ(t),

0, otherwise.(36)

µf,i0 (t) = ηf,i(t)λf , (37)

ϵf,ihf,iℓ

(t) = ϵmin, if Qf,iin,ℓ > Γwℓ. (38)

The proof of the above theorem can be found in AppendixC. Notice that (36) implies that the optimal choice is to routethe input through the path with minimum delay (see Remark2), which matches our intuition. (38) can be interpreted asfollows. When αf,i

ℓ (t) > 0 for all f, i, ℓ, the output queuesare long; therefore we shall use the input queues to buffermore flow, i.e., to deliver as small amount of flow from inputto corresponding output queues as possible. Based on thedefinition of αf,i

ℓ as in (18), the condition αf,iℓ (t) > 0 can

be easily satisfied if ξ is small enough. αf,iℓ (t) becomes non-

positive only when Qf,iin,ℓ(t) is much larger than other queues

in ω1 and ω2. In this case, the solution in the above theorem



8

does not hold any longer. Instead, since the queueing delay inQf,i

in,ℓ(t) becomes a dominant factor, the better choice is to usea larger µf,i

hf,iℓ

to deliver more flow from Qf,iin,ℓ(t) to Qf,i

out,ℓ(t).

C. Discussions

Problem 2 is a linear programming with continuous decisionvariables which is not difficult to solve. The problem scaledepends on the number and lengths of the paths as thenumber of decision variables is |ηf,i(t)| + |ϵf,i

hf,iℓ

(t)| =

|Pf | +∑

i∈Pf |Lf,i|. Moreover, if αf,iℓ (t) > 0 for all f, i, ℓ,

the optimal solution shown in Theorem 3 only requires simplecalculations. In case that Qf,i

in,ℓ > Γwℓ, ∀f, i, ℓ, Ξf,i0,ℓ(t) = 1

and Ξf,i2,ℓ(t) = hf,i

ℓ , which will further simply the solution inTheorem 3. However, for practical implementations, severalissues need to be considered.

1) Convergence time: In each time slot t, the proposed al-gorithm runs iteratively until converging, and the convergencetime is determined by the minimum-consensus algorithm.Providing that the network is connected, the convergence isguaranteed and the maximum convergence time is character-ized by the network diameter [24].

2) Communication overhead: Both Step 1 and 2 incur extracommunication overhead. Consider the links of non-sourcenodes. In Step 1, the information about λf |ℓ ∈ Lf andQf,i

in,ℓ(t), Qf,iout,ℓ(t)|ℓ ∈ Lf,i, which amounts to a total of at

most∑

f∈F (1+2|Pf |) values, is reported from the links to thesource nodes. On average, each link should transmit and alsoforward at most

∑f∈F

∑i∈Fi(1 +

|Lf,i|2 )

∑f∈F (1 + 2|Pf |)

values, where |Lf,i|2 is the average number of hops averaged

over all links along Pf,i. Also, the nodes are involved inexchanging the values of ϵfmax within their neighborhoodsin order to run the minimum-consensus algorithm, whichamounts to a total of at most dmaxLD times of transmissionsper link, where dmax and LD are the maximum node degreeand the network diameter, respectively. In Step 2, each linkshould forward the routing decisions ϵf,ih (t). Similar asabove, the average number of values a link should transmit is∑

f∈F∑

i∈Fi(|Lf,i|

2 +1). Suppose on average transmitting andreceiving a value consume Ecomm amount of energy. Then theaverage communication overhead per link of each non-sourcenode is at most

Ecomm

∑f∈F

∑i∈Fi

(1 +|Lf,i|2

)[1 +∑f∈F

(1 + 2|Pf |)],

which depends on the number of flows and the number andlengths of the paths.

3) Impact of delay: All the above decision-making relatedcommunications also incur some delay of response (i.e., eachlink has to wait some time for the source nodes’ decisionsafter reporting its collected data), which may vary the overallperformance. However, our method can still guarantee networkqueue-length stability if the decision-making related commu-nications can be performed on a separated channel and withoutinterrupting the ongoing data transmissions. Assume networksynchronization is achieved and there is no packet loss. Basedon the path lengths, the maximum delay of all links can be

evaluated. Thus, we can allocate a period of time (at leastequals to the maximum delay) before the end of each timeslot for the links to collect data and conduct the decision-making related communications. Since the data delivery froman input queue to its corresponding output queue can becompleted instantly, the lengths of the input queues will remainunchanged during the allocated period; however, the lengthsof output queues reported to the source nodes may still vary.Since the constraints of Problem 2 involve the lengths of theinput queues but no output queues, the decisions made bythe source nodes with accurate information about the inputqueues can guarantee those constraints and hence guaranteethe network stability according to Theorem 1.

V. SIMULATION RESULTS

In this section, we present simulation results to evaluatethe performance of the proposed joint scheduling and routingmethod. Applying the same distributed maximal schedulingalgorithm, we also compare the proposed distributed delay-aware routing (DDAR), the single-path (SP) and multi-path(MP) routing algorithms proposed in [16] and the cross-layercontrol (CLC) based routing algorithm proposed in [17]. Wedeveloped C++ codes to implement all these algorithms. Sim-ulations are conducted based on a random network topologywith 25 nodes deployed in a 900m×900m area, as shownin Fig. 2. The transmission and interference ranges of eachnode are set to 250m and 500m, respectively. To simulate thechannel diversity, in each slot, the capacity of each link isuniformly generated in the range [0.5, 1.5] with an averageof 1 unit (packets/slot). For a fair comparison, we fix Γ = 1in implementing both DDAR and CLC. As shown in Fig. 2,there are three multi-hop commodity flows to be served bythe network, each of which has the same average input rate λ.To account for flow dynamics, we assume the instantaneousinput varies within [80%, 120%]λ. In the figure, the source anddestination nodes for each flow are marked as s(f) and d(f),respectively. We gradually increase the average flow input rateλ and observe the average per-node backlog (averaged overthose nodes involved in flow transmissions). With a given inputrate λ, i.e., the total amount of input traffic is given, the lagerthe average backlog is, the lower throughput the network canachieve. Also, the average backlog can serve as a measure ofthe network delay performance.

First, we study the performance of the proposed jointscheduling and routing algorithm by examining the averageper-node queues. In Fig. 3, Qout and Qin are the output andinput queue lengths, respectively, averaged over all nodes thatparticipate in the flow transmission, i.e.,

Qin =1

|Ω|∑k∈Ω

∑ℓ:b(ℓ)=k

∑f∈F

∑i∈Pf

Qf,iin,ℓ,

Qout =1

|Ω|∑k∈Ω

∑ℓ:b(ℓ)=k

Qout,ℓ,

where Ω is the set of nodes each of which has at least oneincident link that lies on a candidate path of some flow. Forbetter visibility, we compare Qout and 10Qin. From Fig. 3,



9

0 200 400 600 8000

200

400

600

800

s(0)

d(0)

s(1)

d(1)

s(2)

d(2)

Fig. 2. Network topology.

comparing the two cases (where each node has 2 radios oper-ating with 3 available channels in cases 1 and has 3 radios with5 channels in case 2), we can observe that the performanceis improved by using more radios and channels. As shown inboth figures in Fig. 3, when the per-flow input rate increases,Qout starts to climb quickly when the input rate reaches aturning point (which is 0.35 as in Fig. 3(a) and 0.42 as inFig. 3(b)). Such a turning point indicates that, with a higherinput rate, the network queues Qout,ℓ will become unstable.Therefore, it also indicates the network capacity region underthe distributed maximal scheduling. On the other hand, Qinremains almost stable as the flow input rate increases. A majorreason is that, for each link ℓ, the data transferring from Qf,i

in,ℓ

to Qf,iout,ℓ is conducted within the corresponding tuple instantly

and thus the amount is constrained by neither neighbors’activities nor the scheduling policy. Scrutinizing the curves ofQin, we can find an interesting phenomenon that Qin reaches alocal maximal value at the turning point. Before reaching theturning point, the flow input rate is lower than the capacitythat the scheduling algorithm can support; hence the outputqueue lengths remain low because their buffered packets willbe served with high probabilities. When the input rate equalsthe turning point, the network capacity is reached, and thus thenetwork reaches equilibrium and the input and output queuesare balanced. As the input rate keeps increasing, the networkbecomes more and more congested and the throughput onlyincreases slightly. Thus, the average input queue length Qinincrease a little.

Based on the same distributed maximal scheduling policy,the performance of DDAR, SP, MP and CLC are compared.In DDAR and CLC, the set of candidate paths at the tuple-link level is obtained through the off-line planning technique[17]. In MP, the set of physical link-based candidate pathsis extracted from the above tuple-based path set. The SPalgorithm uses the shortest path for each flow. From Fig. 4, wecan clearly observe that, in both cases, our DDAR algorithmoutperforms the existing ones as the average backlog is alwayslower than those of the others and the turning point lies to theright of the others. As explained in Remark 3, compared withthe other three algorithms, our routing algorithm takes more

0.1 0.2 0.3 0.4 0.50

200

400

600

800

1000

1200

1400

Normalized per-flow rate

Avera

ge p

er-

node q

ueue length

7Qout

10 7Qin

(a) 2 radios and 3 channels

0.1 0.2 0.3 0.4 0.50

200

400

600

800

1000

1200

Normalized per-flow rateAvera

ge p

er-

node q

ueue length

7Qout

10 7Qin

(b) 3 radios and 5 channels

Fig. 3. Performance of the proposed joint scheduling and routing algorithm.

0.1 0.2 0.3 0.4 0.50

100

200

300

400

500

600

700

800


Avera

ge p

er-

node b

acklo

g SP

MP

CLC

DDAR

(a) 2 radios and 3 channels

0.1 0.2 0.3 0.4 0.5 0.60

200

400

600

800


Avera

ge p

er-

node b

acklo

g SP

MP

CLC

DDAR

(b) 3 radios and 5 channels

Fig. 4. Performance comparison among the proposed DDAR, the SP andMP proposed in [16] and the CLC proposed in [17].



10

accurate expectation of flow end-to-end delay into account andcan achieve a better tradeoff between delay and path rate.

3 4 5 60

500

1000

1500

Number of flows

Per

−no

de b

ackl

og

CLC (load=0.3)DDAR (load=0.3)CLC (load=0.4)DDAR (load=0.4)CLC (load=0.5)DDAR (load=0.5)

(a) Under different numbers of flows

25 30 35 40 450

200

400

600

800

1000

1200

Number of nodes

Per

−no

de b

ackl

og

CLC (load=0.3)DDAR (load=0.3)CLC (load=0.4)DDAR (load=0.4)CLC (load=0.5)DDAR (load=0.5)

(b) Under different topologies

Fig. 5. Performance comparison between our proposed method and the CLCmethod proposed in [17].

Both Fig. 4 and the simulation results in [17] have demon-strated that CLC outperforms SP and MP. Therefore, weshall focus on CLC and the proposed DDAR methods in thefollowing. Based on the network topology shown in Fig. 2,we add 1-3 additional flows with randomly chosen source anddestination pairs. As shown in Fig. 5(a), DDAR outperformsCLC in all cases with different number of flows and differentper-flow load. Similarly, we compare the performance ofthe two methods under five randomly generated topologieswith different numbers of nodes but the same number offlows (i.e., 3 flows between randomly chosen source anddestination pairs). The simulation results in Fig. 5(b) alsoadvocates that DDAR is better than CLC. In fact, the routingdecision of CLC is made based on minimizing a delay termΛHf

k +∑

p Qfp(t)Υ

fk,p, where Hf

k is the length of path k

of flow f and Υfk,p indicates whether tuple p (see definition

in Section II) is on the path k. Therefore, the delay tobe minimized in CLC does not include the delay due toscheduling, and the routing delay is expressed in terms ofpath length, which can be very inaccurate. Such differencesin modeling the end-to-end delay along each path can roughlyexplain why DDAR has better performance than CLC.

VI. CONCLUSION

We have proposed a cross-layer framework for studying dis-tributed scheduling and routing in general MR-MC networks.Our scheduling policy is based on the distributed maximalscheduling implemented on the tuple-based MR-MC networkmodel. Under this policy, the network stability is analyzedand a set of sufficient conditions are discovered. A distributeddelay-aware routing method is proposed based on minimizingthe Lyapunov drift. As different from existing distributedrouting methods which only take partial end-to-end delay inmaking routing decisions, our method accounts for all thedelay due to scheduling and queueing in both input and outputqueues of the links. Through extensive simulations, we showedthat the queues maintained at the routing controllers are almoststable as the flow input rate increases. We also showed that theproposed joint scheduling and routing algorithm outperformsexisting SP, MP and CLC algorithms in terms of averagebacklog.

APPENDIX AProof of Theorem 1: First, consider the stability of the

input queues of incident tuple-links of flow source nodes.Based on (3), ∀ℓ ∈ ℓ|∃f ∈ F , b(ℓ) = s(f), we have

E[∆Qf,iin,ℓ(t)|Q

f,iin,ℓ(t)]

, E[Qf,iin,ℓ(t+ 1)−Qf,i

in,ℓ(t)|Qf,iin,ℓ(t)]

= E[Af,i(t)−Df,iℓ (t)|Qf,i

in,ℓ(t)]

= λfηf,i(t)− µf,i

0 (t)1Qf,iin,ℓ(t)>Γwℓ

≤

0, if Qf,iin,ℓ(t) > Γwℓ,

λfηf,i(t), otherwise,

where we have used the condition (10). Thus, the stabilityof Qf,i

in,ℓ(t) is guaranteed in mean sense as the expectationof Qf,i

in,ℓ(t) will decrease or remain unchanged once Qf,iin,ℓ(t)

exceeds Γwℓ.In the second step, consider the Lyapunov function V1(Qout)

as defined in (14). The drift is:

∆V1(t) = V1(Qout(t+ 1))− V1(Qout(t))

=1

2

∑ℓ∈L

Qout,ℓ(t) +Dℓ(t)− Tℓ(t)

wℓ

×

(∑ℓ′∈Iℓ

Qout,ℓ′(t) +Dℓ′(t)− Tℓ′(t)

wℓ′

)

− 1

2

∑ℓ∈L

Qout,ℓ(t)

wℓ

∑ℓ′∈Iℓ

Qout,ℓ′(t)

wℓ′

=∑ℓ∈L

Qout,ℓ(t)

wℓ

∑ℓ′∈Iℓ

Dℓ′(t)− Tℓ′(t)

wℓ′

+1

2

∑ℓ∈L

Tℓ(t)−Dℓ(t)

wℓ

∑ℓ′∈Iℓ


wℓ′

≤∑ℓ∈L

Qout,ℓ(t)

wℓ

∑ℓ′∈Iℓ


wℓ′+ C1 (39)

=∑ℓ∈L

Qout,ℓ(t)

wℓ

∑ℓ′∈Iℓ

(Dℓ′(t)

wℓ′− πℓ′(t)

)+ C1.



11

where C1 ≥ 0 is a constant. Inequality (39) holds becauseboth T f,i

ℓ (t) and Df,iℓ (t) are finite values as constrained by

wℓ and Γ, respectively. For any link ℓ,∑

ℓ′∈Iℓπℓ′ ≥ 0. In

particular, if ℓ is backlogged, we have (8). Thus,

∆V1(t) ≤∑ℓ∈L

Qout,ℓ(t)

wℓ

(∑ℓ′∈Iℓ

Dℓ′(t)

wℓ′− 1

)

−∑

ℓ:Qout,ℓ(t)≤Γwℓ

Qout,ℓ(t)

wℓ

(∑ℓ′∈Iℓ

Dℓ′(t)

wℓ′− 1

)

+∑

ℓ:Qout,ℓ(t)≤Γwℓ

Qout,ℓ(t)

wℓ

∑ℓ′∈Iℓ

Dℓ′(t)

wℓ′+ C1 (40)

≤∑ℓ∈L

Qout,ℓ(t)

wℓ

(∑ℓ′∈Iℓ

Dℓ′(t)

wℓ′− 1

)+ C2,

where C2 ≥ 0 is another constant. Furthermore, based on (1),

E[∆V1(t)|Qout(t)]

≤∑ℓ∈L

Qout,ℓ(t)

wℓ

∑ℓ′∈Iℓ

∑f∈F

∑i∈Pf

µf,i

hf,i

ℓ′(t)1Qf,i

in,ℓ′ (t)>Γwℓ′

wℓ′− 1

+ C2 (41)

≤ − (1− amax)∑ℓ∈L

Qout,ℓ(t)

wℓ+ C2, (42)

where the last inequality holds due to condition (9). Therefore,based on the Lyapunov drift theory [25], Qout is stable in meansense.

Finally, to prove the stability of Qin,N , consider the Lya-punov function V (Qin,N ,Qout) as defined in (13). Similar to(39),

∆V2(t) = V2(Qin,N (t+ 1))− V2(Qin,N (t))

≤∑ℓ∈L

∑f∈F

∑i∈Pf


in,Nf,iℓ

(t)

wℓ

Df,iℓ (t)−Df,i

Nf,iℓ

(t)

wℓ

+ C3,

≤ C3 +∑ℓ∈L

∑f∈F

∑i∈Pf


in,Nf,iℓ

(t)

w2ℓ

×

(µf,i

hf,iℓ


− µf,i

hf,iℓ +1

(t)1Qf,i

in,Nf,iℓ

(t)>ΓwN

f,iℓ

)(43)

where C3 is a constant. Above we have used the fact that

hf,i

Nf,iℓ

= hf,iℓ + 1. Thus,

E[∆V2(t)|Qin,N (t),Qout(t)]

≤ C3 +∑ℓ∈L

∑f∈F

∑i∈Pf

µf,i

hf,iℓ

(t)

w2ℓ

Qf,iout,ℓ(t)

+∑ℓ∈L

∑f∈F

∑i∈Pf ,Qf,i

in,ℓ(t)>Γwℓ

Qf,i

in,Nf,iℓ

(t)>ΓwN

f,iℓ

Qf,i

in,Nf,iℓ

(t)

w2ℓ

×(µf,i

hf,iℓ

(t)− µf,i

hf,iℓ +1

(t))

+∑ℓ∈L

∑f∈F

∑i∈Pf ,Qf,i

in,ℓ(t)>Γwℓ

Qf,i

in,Nf,iℓ

(t)≤ΓwN

f,iℓ

Qf,i

in,Nf,iℓ

(t)

w2ℓ

µf,i

hf,iℓ

(t)

−∑ℓ∈L

∑f∈F

∑i∈Pf ,Qf,i

in,ℓ(t)≤Γwℓ

Qf,i

in,Nf,iℓ

(t)>ΓwN

f,iℓ

Qf,i

in,Nf,iℓ

(t)

w2ℓ

µf,i

hf,iℓ +1

(t) (44)

≤ C3 +∑ℓ∈L

∑f∈F

∑i∈Pf

µf,i

hf,iℓ

(t)

w2ℓ

Qf,iout,ℓ(t)

− ϵmin

∑ℓ∈L

∑f∈F

∑i∈Pf ,Qf,i

in,Nf,iℓ

(t)>ΓwN

f,iℓ

Qf,i

in,Nf,iℓ

(t)

w2ℓ

+∑ℓ∈L

∑f∈F

∑i∈Pf

Qf,i

in,Nf,iℓ

(t)

w2ℓ

µmax(t) (45)

≤∑ℓ∈L

∑f∈F

∑i∈Pf

µf,i

hf,iℓ

(t)

w2ℓ

Qf,iout,ℓ(t)−

ϵmin

w2ℓ

Qf,i

in,Nf,iℓ

(t)

+ C4

≤∑ℓ∈L

µmax

w2ℓ

Qout,ℓ(t)−∑ℓ∈L

∑f∈F

∑i∈Pf

ϵmin

ρmaxw2ℓ

Qf,i

in,Nf,iℓ

(t) + C4,

where

ρmax , maxf,i,ℓ

w2Nf,i

ℓ

w2ℓ

,

µmax(t) , maxf,i,ℓ

µf,i

hf,iℓ

(t)= max

f,i

µf,i|Lf,i|−1

(t).

In the above, (45) holds due to conditions (11) and (12).Combining the above inequality with (42), we get

E[∆V (t)|Qin,N (t)]

≤ − E

[∑ℓ∈L

1

wℓ

(1− amax − ξ

µmax(t)

wℓ

)Qout,ℓ

]− ξ

∑ℓ∈L

∑f∈F

∑i∈Pf

ϵmin

ρmaxw2ℓ

Qf,i

in,Nf,iℓ

(t) + C2 + C4

≤ − ξ∑ℓ∈L

∑f∈F

∑i∈Pf

ϵmin

ρmaxw2ℓ

Qf,i

in,Nf,iℓ

(t) + C2 + C4,



12

The last inequality holds because of the stability of Qout whichhas been proven above. Let ξ < (1−amax)wℓ

µmax, we will have

1−amax−ξ µmax

wℓ> 0. Therefore, the Lyapunov function V (t)

will be negative if Qf,i

in,Nf,iℓ

is large enough, which implies

that every Qf,i

in,Nf,iℓ

is stable in mean sense. Thus, the wholetheorem is proven.

APPENDIX B

Proof of Theorem 2: It is suffice to prove that for anyλ such that Kλ ∈ Λopt, λ can stabilize the network under theproposed scheduling algorithm. Since the network is stableunder Kλ, based on the definition of network interferencedegree, a necessary condition is that, ∀ℓ ∈ L,∑

ℓ′∈Iℓ

∑f∈F

Kλf

wℓ′1ℓ′∈Lf < K

⇒∑ℓ′∈Iℓ

∑f∈F

λf

wℓ′1ℓ′∈Lf < 1. (46)

With λ satisfies the above inequality, we are to construct theconditions in Theorem 1 such that the network is stable underλ and the proposed scheduling algorithm. ∀f, i and ℓ such thathf,iℓ < |Lf,i| − 1, let µf,i

0 (t) = λfηf,i(t) which guarantees

condition (10) and

µf,i

hf,iℓ +1

(t)− µf,i

hf,iℓ

(t) = ϵf,i(t).

Thus, along the path Pf,i, we have

µf,ih (t) = µf,i

0 (t) + hϵf,i(t),

where h ∈ 0, 1, . . . , |Lf,i| − 1 is the number of hops awayfrom the source tuple. Then, ∀ℓ ∈ L,

∑ℓ′∈Iℓ

∑f∈F

∑i∈Pf

µf,i

hf,i

ℓ′(t)

wℓ′

≤∑ℓ′∈Iℓ

∑f∈F

∑i∈Pf

λfηf,i(t) + (|Lf,i| − 1)ϵf,i(t)

wℓ′

≤ maxℓ

∑ℓ′∈Iℓ

∑f∈F

λf

wℓ′1ℓ′∈Lf

+∑ℓ′∈Iℓ

∑f∈F

∑i∈Pf

(|Lf,i| − 1)ϵf,i(t)

wℓ′1ℓ′∈Lf,i .

In the right-hand-side of the above formula, the first term isless than 1 according to (46), while the second term can becontrolled to any small value by choosing proper ϵmin > 0 andϵf,i(t)|ϵf,i(t) ≥ ϵmin. Therefore, there exists amax ∈ (0, 1)and ϵmin > 0 such that the conditions (9)-(12) in Theorem 1are satisfied.

APPENDIX C

Proof of Theorem 3: For the links ℓ|Qf,iin,ℓ(t) ≤ Γwℓ,

we already discussed that the decisions will be µf,i

hf,iℓ

(t) = 0.

For the rest links, if αf,iℓ (t) > 0, their µf,i

hf,iℓ

(t) should bechosen as small as possible in order to minimize the objective

functions in both Problem 1 and Problem 2. Thus, we obtain(37) and (38) for both Problems. Then, for each flow f , theobjective function of Problem 2 reduces to∑

i∈Pf

∑ℓ∈Lf,i

Jf,iℓ (t)

=∑i∈Pf

∑ℓ∈Lf,i

αf,iℓ (t)

wℓ

(µf,i0 (t)Ξf,i

0,ℓ(t) + ϵminΞf,i2,ℓ(t)

)= λf

∑i∈Pf

ηf,i(t)∑

ℓ∈Lf,i

αf,iℓ (t)

wℓΞf,i0,ℓ(t)

+ ϵmin

∑i∈Pf

αf,iℓ (t)

wℓΞf,i2,ℓ(t)

≥ λf mini∈Pf

∑ℓ∈Lf,i

αf,iℓ (t)

wℓΞf,i0,ℓ(t) + ϵmin

∑i∈Pf

αf,iℓ (t)

wℓΞf,i2,ℓ(t),

where the equality in the last line holds if (36) is satisfied.Since (36) can be obtained for each flow, it is also thesolution for Problem 1. In addition, with (36)-(38), it is easy toverify that both conditions in (9) and (31) are satisfied, whichcompletes the proof of Theorem 3.

REFERENCES

[1] M. Kodialam and T. Nandagopal, “Characterizing the capacity regionin multi-radio multi-channel wireless mesh networks,” in Proc. ACMMobicom, 2005, pp. 73–87.

[2] M. Alicherry, R. Bhatia, and L. E. Li, “Joint channel assignmentand routing for throughput optimization in multi-radio wireless meshnetworks,” in Proc. ACM Mobicom, 2005, pp. 58–72.

[3] N. Chakchouk and B. Hamdaoui, “Traffic and interference awarescheduling for multiradio multichannel wireless mesh networks,” IEEETrans. Veh. Technol., vol. 60, no. 2, pp. 555–565, 2011.

[4] J. Chen, Q. Yu, P. Cheng, Y. Sun, Y. Fan, and X. Shen, “Game theo-retical approach for channel allocation in wireless sensor and actuatornetworks,” IEEE Trans. Autom. Control, vol. 56, no. 10, pp. 2332–2344,2011.

[5] Y. Cheng, H. Li, and P.-J. Wan, “A theoretical framework for optimalcooperative networking in multiradio multichannel wireless networks,”IEEE Wireless Commun., vol. 19, no. 2, pp. 66–73, 2012.

[6] Y. Qin, J. Zheng, X. Wang, H. Luo, H. Yu, X. Tian, and X. Gan,“Opportunistic scheduling and channel allocation in MC-MR cognitiveradio networks,” IEEE Trans. Veh. Technol., vol. 63, no. 7, pp. 3351–3368, 2014.

[7] Y. Long, H. Li, M. Pan, Y. Fang, and T. F. Wong, “A fair QoS-awareresource-allocation scheme for multiradio multichannel networks,” IEEETrans. Veh. Technol., vol. 62, no. 7, pp. 3349–3358, 2013.

[8] L. Bui, R. Srikant, and A. Stolyar, “Novel architectures and algorithmsfor delay reduction in back-pressure scheduling and routing,” in Proc.IEEE INFOCOM, 2009, pp. 2936–2940.

[9] L. Ying, R. Srikant, D. Towsley, and S. Liu, “Cluster-based back-pressure routing algorithm,” IEEE/ACM Trans. Netw., vol. 19, no. 6,pp. 1773–1786, 2011.

[10] C. Joo, X. Lin, and N. B. Shroff, “Understanding the capacity regionof the greedy maximal scheduling algorithm in multihop wirelessnetworks,” IEEE/ACM Trans. Netw., vol. 17, no. 4, pp. 1132–1145, 2009.

[11] S. Bodas, S. Shakkottai, L. Ying, and R. Srikant, “Low-complexityscheduling algorithms for multichannel downlink wireless networks,”IEEE/ACM Trans. Netw., vol. 20, no. 5, pp. 1608–1621, 2012.

[12] J. Chen, W. Xu, S. He, Y. Sun, P. Thulasiraman, and X. S. Shen,“Utility-based asynchronous flow control algorithm for wireless sensornetworks,” IEEE J. Sel. Areas Commun., vol. 28, no. 7, pp. 1116–1126,2010.

[13] A. Gupta, X. Lin, and R. Srikant, “Low-complexity distributed schedul-ing algorithms for wireless networks,” IEEE/ACM Trans. Netw., vol. 17,no. 6, pp. 1846–1859, 2009.

[14] P. Chaporkar, K. Kar, X. Luo, and S. Sarkar, “Throughput and fairnessguarantees through maximal scheduling in wireless networks,” IEEETrans. Inf. Theory, vol. 54, no. 2, pp. 572–594, 2008.



13

[15] X. Wu, R. Srikant, and J. R. Perkins, “Scheduling efficiency ofdistributed greedy scheduling algorithms in wireless networks,” IEEETrans. Mobile Comput., vol. 6, no. 6, pp. 595–605, 2007.

[16] X. Lin and S. B. Rasool, “Distributed and provably efficient algorithmsfor joint channel-assignment, scheduling, and routing in multichannelad hoc wireless networks,” IEEE/ACM Trans. Netw., vol. 17, no. 6, pp.1874–1887, 2009.

[17] Y. Cheng, H. Li, D. M. Shila, and X. Cao, “A systematic studyof maximal scheduling algorithms in multiradio multichannel wirelessnetworks,” IEEE/ACM Trans. Netw., vol. 23, no. 4, pp. 1342–1355, 2015.

[18] B. Ji, C. Joo, and N. B. Shroff, “Exploring the inefficiency and instabilityof back-pressure algorithms,” in Proc. IEEE INFOCOM, 2013, pp.1528–1536.

[19] L. Ying, S. Shakkottai, A. Reddy, and S. Liu, “On combining shortest-path and back-pressure routing over multihop wireless networks,”IEEE/ACM Trans. Netw., vol. 19, no. 3, pp. 841–854, 2011.

[20] B. Ji, G. R. Gupta, X. Lin, and N. B. Shroff, “Low-complexity schedul-ing policies for achieving throughput and asymptotic delay optimalityin multichannel wireless networks,” IEEE/ACM Trans. Netw., vol. 22,no. 6, pp. 1911–1924, 2014.

[21] D. Xue and E. Ekici, “Delay-guaranteed cross-layer scheduling inmultihop wireless networks,” IEEE/ACM Trans. Netw., vol. 21, no. 6,pp. 1696–1707, 2013.

[22] H. Li, Y. Cheng, C. Zhou, and W. Zhuang, “Minimizing end-to-enddelay: a novel routing metric for multi-radio wireless mesh networks,”in Proc. IEEE INFOCOM, 2009, pp. 46–54.

[23] V. Yadav and M. V. Salapaka, “Distributed protocol for determiningwhen averaging consensus is reached,” in Proc. Annual Allerton Conf.,2007, pp. 715–720.

[24] H. Liu, X. Cao, J. He, P. Cheng, C. Li, J. Chen, and Y. Sun, “Distributedidentification of the most critical node for average consensus,” IEEETrans. Signal Process., vol. 63, no. 16, pp. 4315–4328, 2015.

[25] M. J. Neely, E. Modiano, and C. E. Rohrs, “Dynamic power allocationand routing for time-varying wireless networks,” IEEE J. Sel. AreasCommun., vol. 23, no. 1, pp. 89–103, 2005.

Xianghui Cao (IEEE S’08-M’11) received the B.S.and Ph.D. degrees in control science and engineeringfrom Zhejiang University, Hangzhou, China, in 2006and 2011, respectively. From December 2007 to June2009, he was a Visiting Scholar with the Departmentof Computer Science, The University of Alabama,Tuscaloosa, AL, USA. From July 2012 to July2015, he was a Senior Research Associate with theDepartment of Electrical and Computer Engineering,Illinois Institute of Technology, Chicago, IL, USA.Currently he is an associate professor with the

School of Automation, Southeast University, Nanjing, China. His researchinterests include wireless sensor and actuator networks, wireless networkperformance analysis and network security. He serves as Publicity Co-chairfor ACM MobiHoc 2015, Symposium Co-chair for IEEE/CIC ICCC 2015,and TPC Member for a number of conferences, including IEEE Globecom,IEEE ICC, and IEEE VTC. He also serves as an Associate Editor of severaljournals, including KSII Transactions on Internet and Information Systems,Security and Communication Networks (Wiley), and the International Journalof Ad Hoc and Ubiquitous Computing. He was a recipient of the Best PaperRunner-Up Award from ACM MobiHoc 2014.

Lu Liu (IEEE S’13) received the B.S. degreein Automation from Tsinghua University, Beijing,China, in 2010, and the M.S. degree in ElectricalEngineering from Illinois Institute of Technology,Chicago, IL, USA, in 2012. She is currently pursuingthe Ph.D. degree in the Department of Electricaland Computer Engineering at Illinois Institute ofTechnology, Chicago, IL, USA. Her current researchinterests include energy efficient networking andcommunications, performance analysis and protocoldesign of wireless networks.

Wenlong Shen (IEEE S’13) received the B.E. de-gree in Electrical Engineering from Beihang Uni-versity, Beijing, China, in 2010, and the M.S. degreein Telecommunication from University of Maryland,College Park, USA, in 2012. He is currently pursu-ing the Ph.D. degree in the department of Electricaland Computer Engineering, Illinois Institute of Tech-nology, Chicago, IL, USA. His research interestsinclude vehicular ad hoc networks, mobile cloudcomputing, and network security.

Yu Cheng (IEEE S’01-M’04-SM’09) received theB.E. and M.E. degrees in Electronic Engineeringfrom Tsinghua University, Beijing, China, in 1995and 1998, respectively, and the Ph.D. degree inElectrical and Computer Engineering from the Uni-versity of Waterloo, Waterloo, Ontario, Canada, in2003. From September 2004 to July 2006, he wasa postdoctoral research fellow in the Departmentof Electrical and Computer Engineering, Universityof Toronto, Ontario, Canada. Since August 2006,he has been with the Department of Electrical and

Computer Engineering, Illinois Institute of Technology, Chicago, Illinois,USA, where he is now an Associate Professor. His research interests includenext-generation Internet architectures and management, wireless networkperformance analysis, network security, and wireless/wireline interworking.He received a Best Paper Award from the conferences QShine 2007 and IEEEICC 2011, and the Best Paper Runner-Up Award from ACM MobiHoc 2014.He received the National Science Foundation (NSF) CAREER AWARD in2011 and IIT Sigma Xi Research Award in the junior faculty division in 2013.He served as a Co-Chair for the Wireless Networking Symposium of IEEEICC 2009, a Co-Chair for the Communications QoS, Reliability, and ModelingSymposium of IEEE GLOBECOM 2011, a Co-Chair for the Signal Processingfor Communications Symposium of IEEE ICC 2012, a Co-Chair for the AdHoc and Sensor Networking Symposium of IEEE GLOBECOM 2013, anda Technical Program Committee (TPC) Co-Chair for WASA 2011, ICNC2015, and IEEE/CIC ICCC 2015. He is a founding Vice Chair of the IEEEComSoc Technical Subcommittee on Green Communications and Computing.He is an Associated Editor for IEEE Transactions on Vehicular Technologyand the New Books & Multimedia Column Editor for IEEE Network. He isa senior member of the IEEE.

Documents

Distributed Scheduling and Delay-Aware Routing in Multi ...yucheng/YCheng_TVT16_2.pdfSC networks into tuple-based MR-MC networks. By analyzing the drift of well-deﬁned Lyapunov functions,