Embed Size (px)
Citation preview
1
A Distributed Algorithm for Max-min FairBandwidth SharingXinran (Ryan) Wu and Andrew A. Chien
Department of Computer Science and Engineering andCenter for Networked Systems
University of California, San DiegoLa Jolla, CA, 92093-0114, USA
Abstract— Dense Wavelength Division Multiplexing (DWDM),dedicated optical paths, high speed switches, and high speedrouters are giving rise to networks with plentiful bandwidthin the core. In such networks, bottlenecks and congestion areconcentrated move at the edge and at end nodes. We considerhow to efficiently and fairly share source and sink capacities insuch network environments.
As a foundation, we first describe a global algorithm whichcalculates max-min fair rate allocation for network sessions withknown desired peak rates. We then present a novel distributedrate allocation algorithm which controls each source and sinkindependently with local information, adaptively allocating itscapacity to candidate sessions. This algorithm is given no knowl-edge of the desired session rates, but rather discovers them. Thisallows it to be used in general network settings. We prove thatour distributed algorithm converges rapidly to the max-min fairrate allocation in a discrete time system. Simulations confirmthis behavior, and further show that convergence is in practicefast in networks of 64 to 1024 nodes. Simulations also prove thatthe stability and convergence results still hold in asynchronousdistributed settings with a range of network delays.
I. INTRODUCTION
Continuing advances in optical transmission are drivingrapid increases in feasible network bandwidths at densities ofover a terabit per optical fiber for both metro and long haul net-works. A key driver is dense wavelength division multiplexing(DWDM) which allows each optical fiber to carry hundredsof lambdas (i.e. wavelengths) of 10 or 40 Gbps. These capa-bilities are being used to build high-speed traditional shared,routed internet networks, private dark fiber networks [39], [23],and based on new technologies for dynamic configuration,dynamic private lambda networks. Examples of these dynamicprivate networks include LambdaGrid [18] , OptIPuter [45],CANARIE [3], Netherlight [8], NAREGI [7], etc.).
Our research focuses in particular on networks where setsof high-speed private optical paths are constructed dynami-cally, forming a high-performance private network. In suchnetworks, the network bandwidth in the core of lambdas ishigh, matching or exceeding the speeds at which endpoints cansource or sink network traffic. In short, the network of lambdasprovides high bandwidth, very low loss packet communication.We call these high speed, private networks “lambda networks.”
However, lambda-networks differ from traditional sharedIP networks in several key ways. First, shared IP networkswere designed to manage internal resource bottlenecks (links,
routers or switches), and include mechanisms for efficientlink multiplexing and management of network congestion. Incontrast, lambda-networks have abundant network bandwidthand switching, and thus have little network congestion andlittle packet loss. However, congestion does arise at end nodesdue to their limited capacity, and give rise to packet loss there.Second, because of application trends in WWW technologiesand other large-scale data sharing, there is increasing use ofmultipoint-to-point traffic patterns (as evidenced in the world-wide web, P2P networks [6], [2], and large-scale scientific datasharing [4], [5], [1]). High-speed core networks enable manyfast flows to converge on an endpoint, raising challenges inefficient and fair sharing of source and sink capacity.
As a result, we study how end nodes should manage theircapacity across multiple sessions in networks with plentifulbandwidth. In any distributed networks, end nodes must allo-cate their capacity based on incomplete information: local ses-sion status and approximations of desired rates. This problemdiffers in three important ways from that addressed by TCPand its variants [31], [21], [22], [49]. First, TCP and variantsall manage single flows, providing rate and congestion control.They do not consider interactions amongst flows. Second, TCPand variants generally assume shared networks with internalcongestion, so they focus on managing congestive packet losswithin the network, not at the endpoints. Third, when TCPflows share a link with different round-trip delays (RTT), thedelivered bandwidth is inversely proportional to RTT, so thesharing is unfair, and this is not considered a major problem.In contrast, achieving fair allocation with varied RTT’s is ourprimary goal.
Our distributed rate allocation problem is distinct from thatexplored in ATM networks [27], [14] and IP networks [35],which focus primarily on allocating router and switch capacitywithin the network. We solve the inverse problem. With plen-tiful bandwidth in the core, we focus on resource allocationfor edge nodes.
Our work on distributed rate allocation is motivated by andprovides an algorithmic basis for an increasing number of newprotocols [11], [38], [48]. The analytical and dynamic behaviormodeling presented in this paper complement the wealth ofempirical work on these protocols. For example, considerthe Group Transport Protocol (GTP) [48] which receiver-based rate control and coordinates amongst active sessions toachieve high performance in lambda networks. Our previous
2
work involved a range of empirical studies, but the distributedrate allocation framework described in this paper significantlyadvances our previous GTP work, extending management andcontrol to source and sinks and providing strong properties ofefficiency, fairness, and convergence.
The critical objectives for distributed rate allocation innetworks are:• Efficiency: Maximize utilization of source and sink ca-
pacity;• Fairness: Allocate a fair share of source and sink capacity
to each session (eventually);• Stability & Convergence: Converge to a unique, optimal,
and fair rate allocation from any initial state.We describe a discrete-time distributed rate allocation algo-
rithm. The distributed algorithm uses local information onlyto allocate capacity at each source and sink, assigning anexpected rate to each session. So for each session, its actualrate is determined by the minimum of the expected rate at itssource and sink, and its desired rate (which is not generallyknown to the network). Our distributed algorithm convergesrapidly to a max-min fair rate allocation [12] [34], [27], [42],and also adapts quickly when desired rates change.
Specific contributions are as follows:1) a mathematical model of lambda-networks, and a for-
mulation of the bandwidth sharing problem in lambda-networks,
2) a global rate allocation algorithm which calculates theoptimum rate allocation given the constraints of a set offinite desired session rates. We prove that the algorithmconverges to the unique max-min fair rate allocation,
3) a distributed rate allocation algorithm which allocatessource and sink bandwidth. We prove that the distributedalgorithm converges to the unique max-min fair rateallocation,
4) an evaluation of the distributed rate allocation algorithmwhich shows that it achieves efficient and fair alloca-tions. And further that it converges rapidly for a rangeof network configurations,
5) an evaluation of the distributed rate allocation algorithmin an asynchronous, distributed network (varying size,latency, and synchrony). Our simulation results demon-strate the strong convergence and stability properties alsohold in these more practical circumstances.
This paper describes work that is part of the OptIPuterproject [45], [46], [47], [48], which explores the use ofconfigurable optical networks to provide high bandwidth andtighter coupling amongst computational and storage resource.
The rest of the paper is organized as follows. In SectionII, we introduce the model of the lambda-networks, andformulate the bandwidth sharing and rate adaptation problemin the lambda-networks. In Section III, we present a globalalgorithm which calculates the max-min fair rate allocationin the lambda-networks. We prove that such max-min fairallocation is unique. In Section IV, we propose a distributedalgorithm with capacity allocation and rate adaptation at eachsource and sink. We prove the stability and convergenceproperties of the proposed algorithm. In Section V and VI,
we conduct numerical case studies to illustrate the convergenceproperties of the distributed algorithm in both synchronous andasynchronous distributed settings. Related works are brieflydescribed in Section VII, and we conclude in Section VIII.
II. PROBLEM FORMULATION
A. Modeling Lambda-Networks
In the lambda-networks, high-speed private optical paths canbe constructed on-demand to provide plentiful dedicated end-to-end bandwidth. Such networks differ from the traditionalshared Internet in that there is essentially no packet loss dueto congestion within the network, and any contention andcongestion loss happens at end nodes (sources and sinks).
We make the following assumptions when modelinglambda-networks.
1) With plenty of network bandwidth is provided by pri-vate, dynamically configured lambdas, there is no trafficcongestion within the network. Because the capacity oflambda-network is higher than the desired rate of eachsession, no modeling of network internals is required.The rate control and allocation problem is reduced tothe problem of how to efficiently and fairly share thesource and sink capacities amongst the traffic sessions.
2) Each each source and sink node has explicit knowl-edge regarding its capacity, usually its network interfacespeed, local packet processing capacity or a shared linknearby. With capacity information, each node knowshow much capacity it has to allocate across its sessions.
3) each session has its own desired rate – the peak rate itcan reach. The desired rate can be also interpreted asthe maximum data handling speed of the applicationsusing the network. The desired rate of each session isunknown to its source and sink, and can vary over time.
B. Notation
Our notation for modeling lambda-networks follows.Consider a lambda-Network G with a finite set V of nodes
and a set K of K active sessions. Let VS and VR denote theset of source and sink nodes, respectively. Note that multiplesessions may start or terminate at the same source or sinknode.
Consider a discrete-time model, where time is divided intoslots of equal length. Let each session k ∈ K be associatedwith a desired (peak) rate Mk. Let xk(t) be the instant rate ofsession k in time slot t. Let x(t) = (x1(t), x2(t), ..., xK(t))be the rate vector of all active sessions in time slot t.
Let each source and sink v ∈ V have a capacity Cv . Itis either the sending or receiving capacity of node v, whenv ∈ VS or v ∈ VR, respectively. Let Kv denote the set ofsessions that node v participates in. Note that in our modeleach node v is either a sink or a source node, but not both.
Therefore G(V, C,K,M) characterizes an instance of thebandwidth sharing problem in the lambda-networks.
3
C. Rate Allocation Problem
The challenge of bandwidth sharing and rate allocation ina lambda network is how to efficiently and fairly share thecapacity of each source and sink among active sessions. Ofcourse, the allocation algorithm should also be stable.
We describe the main bandwidth sharing objectives asfollows.
First, the rate allocation (bandwidth sharing) algorithmshould efficiently utilize of the capacity of each source andsink while maintaining feasibility. Feasibility, simply put,requires that aggregate rate of the sessions for each sourceand sink does not exceed its capacity, and each session’s ratedoes not exceed its desired rate. Formally, feasibility can bedefined as follows.
Definition 1: Feasibility. A rate vector x is feasible, whenfor each node v ∈ V ,
∑
k∈Kv
xk ≤ Cv,
and for each session k ∈ K we have that xk ≤ Mk.Second, the a rate allocation algorithm should treat all active
sessions fairly. We adopt Max-min Fairness [12] as the criteria,as it is widely-used in wired and wireless networks [34], [42],[41]. Max-min fairness maximizes the rates of the sessionswith lower rates, and shares the remaining bandwidth untilthe network is fully utilizes. Formally, max-min fairness isdefined as follows.
Definition 2: Max-min Fairness. A feasible rate vector xis max-min fair if it is not possible to increase the rate of asession xp while maintaining feasibility, without reducing therate of some session xq (p 6= q) with xp ≤ xq . i.e. for anyother feasible vector y,
(∃p ∈ K)yp > xp =⇒ (∃q ∈ K)yq < xq < xp.Third, the distributed rate allocation algorithm should con-
verge rapidly, reaching a unique rate allocation which is max-min fair. Consider under a discrete-time system, in time slott, the rate vector x(t) is updated as
x(t + 1) = f(x(t)),
where f(·) is the update function. The function f(·) maydepend on session desired rates, the status of each sourceand sink, etc. For good network behavior, the rate allocationalgorithm should be stable and converge.
Definition 3: Stability and Convergence. An algorithm isstable if there is a unique equilibrium rate vector x∗ =(x∗1, x
∗2,..., x
∗K), s.t.
x∗ = f(x∗).
An algorithm converges, if when all K sessions are long-lived, the rate vector (x1, x2,..., xK) over time approaches andachieves the unique equilibrium rate allocation x∗, regardlessof its initial state, session starting sequence, or other temporaldetails.
The definition of stability is straightforward. The definitionof convergence not only encompasses arbitrary initial state, butalso changes in session desired rates during the convergenceprocess. In the following section, we study several global anddistributed bandwidth allocation algorithms.
III. GLOBAL ALGORITHM
A global algorithm is presented to calculate the max-minfair allocation. We show that the result rate vector is the uniquemax-min fair allocation for the bandwidth sharing problem wedefined in the previous section.
A. Global Algorithm
With global knowledge, max-min fair rates can be calculatedin straightforward fashion [12]. At each step the algorithmassigns equal rate shares to sessions at the current bottlenecknode. The bottleneck node is the one with with the minimumaverage rate after its remaining capacity is assigned to theremaining sessions (also known as undetermined sessions).Formally,
Definition 4: Bottleneck Node. Given a partially deter-mined rate vector x, node v ∈ V is called the bottleneck node,if we have undetermined sessions k ∈ {j|xj = 0, j ∈ K}, andremaining available capacity at node v divided by the numberof undetermined sessions, xf (also known as the bottleneckrate), satisfies
xf =Cv −∑
i∈Kv xi
|{j|xj = 0, j ∈ Kv}| = minv′∈V
Cv′ −∑i∈Kv′ xi
|{j|xj = 0, j ∈ Kv′}| ,(1)
where node v′ ∈ V such that |{i|xi = 0, i ∈ Kv′}| > 0.Our algorithm extends the classic max-min algorithm [12]
by considering the session’s desired rate.Sessions whose desired rate has already been satisfied are
considered determined, and should thereby be excluded fromthe undetermined sessions.
The global algorithm is formalized as follows.Algorithm 1: Global AlgorithmNotations:x∗: The result rate vector, and x∗ = (x∗1, ..., x
∗K).
b(v): Computes the average rate of undetermined sessionsat node v ∈ V , denoted by
b(v) =Cv −∑
i∈Kv x∗i|{j|x∗j = 0, j ∈ Kv}| . (2)
If there are no undetermined sessions at node v ∈ V , setb(v) = +∞.
1. set x∗ = 0;2. repeat3 if minv∈Vb(v) < mink∈K,x∗
k=0Mk
4 v = arg minv′∈V b(v′);5 ∀ k ∈ Kv , if x∗k = 0 then let x∗k = b(v);6 else7 k = arg mink′∈K,x∗
k′=0Mk′ ;8 x∗k = Mk;9 endif10 until |{j|x∗j = 0, j ∈ K}| = 0.Note that there may be multiple bottleneck nodes with the
same remaining capacity for any given iteration. In that case,we pick an arbitrary node.
4
B. Properties of Global Algorithm
We show that the global algorithm does in fact compute themax-min fair rate allocation.
We show in Proposition 1 that the rate vector x∗ calculatedby Algorithm 1 is the unique max-min fair rate allocation.
Proposition 1: 1. Algorithm 1 (Global Algorithm) termi-nates within K iterations.
2. The rate allocation x∗ computed by Algorithm 1 is max-min fair and unique.
The proofs of the Proposition 1 are straightforward andincluded in Appendix A.
While the global algorithm is simple and has numerousdesirable properties, it has a number of important limitationswhich we relax in subsequent sections. First, the globalalgorithm requires global information for the entire network toset the allocation for each node and session. Second, the globalalgorithm requires desired session rates, which is typicallydifficult to extract from applications. We consider a distributedalgorithm which overcomes these limitations in the followingsection.
IV. DISTRIBUTED ALGORITHM
We present a distributed algorithm which overcomes themajor limitations of the global algorithm. The distributed al-gorithm requires only local node and actual session behavior toproduce max-min fair rate allocation for arbitrary workloads.
A. Distributed Rate Allocation and Adaptation
Consider a discrete-time view where at time t each nodev ∈ V knows the current sending rate xk(t) of each sessionk ∈ Kv , but no knowledge about its desired rate Mk. As aresult, a source and sink cannot tell whether a session withsmall rate in time slot t is constrained by the session’s desiredrate or the capacity of the peer node (complementary source orsink). This means that a static rate allocation scheme (withoutrate adaptation) has no means to determine or adapt to changesin the desired session rate.
To meet these needs, we formulate the rate allocationproblem as a feedback-based rate adaptation problem withcoordinated rate allocation at each source and sink node.Specifically, in each time slot, all source and sink nodesoperate autonomously. In time slot t each sink node allocatesits capacity and sets expected rates x̂r
k for each of its sessions,using the status of all sessions terminating at the same sink:
x̂r(t + 1) = g(x(t)).
The expected rate for each session k, x̂rk(t + 1) at the sink
is then fed back to the source. With this information, eachsource node allocates its capacity to each session in similarfashion. Each source calculates the expected rate x̂s
k for eachof its sessions k ∈ Kv:
x̂s(t + 1) = h(x(t)).
As the network operates, the actual rate of session k ∈ K intime slot t+1 is then determined by the minimum of expected
source rate x̂s(t + 1), expected sink rate x̂r(t + 1), and thesession desired rate Mk, formally
xk(t + 1) = min{x̂s(t + 1), x̂r(t + 1),Mk}. (3)
B. Source and Sink Algorithms
The idea in the distributed algorithm is to adapt eachsession’s current rate towards the max-min fair rate for thesession. More specifically, at each step, each node picksthe session with the minimum rate among its undeterminedsessions and assigns expected rates based on current expectedrate and the target rate. Intuitively, the target rate approximatethe max-min fair allocation of the remaining capacity over theundetermined sessions.
Because expected session rates are calculated one at a time,target rates may vary across even similar sessions. If thecurrent session rate is less than the target rate xf , we adapt ittoward the target rate, as
x̂k(t + 1) = xk(t) + α(xf − xk(t)),
where α ∈ (0, 1) is the adaptation step size (0 < α < 1),and xf is the current remaining capacity over the number ofundetermined sessions:
xf =Cv −∑
i∈Kv,x̂i(t+1) 6=0 xi(t),
|{j|x̂j(t + 1) = 0, j ∈ Kv}| . (4)
We update xf each iteration to reflect the changes inremaining capacity. When the minimum session rate of allundetermined sessions is greater than the target rate xf , wedistribute the remaining bandwidth evenly to the sessions,assigning each of them the target rate, as
x̂k(t + 1) = xf , if xk(t) ≤ xf .
In general, we try to maximize the session rates for sessionswith lower desired rates by considering them earlier. Sessionswith minimum rates are considered first, and their ratescontinue to increase as long as their rate is still minimum.It only stops increasing when it is restricted by its peer nodeor desired rate, or reaches the target rate. In the last case,the target rate is actually Cv/|Kv|, a fair share of bandwidthwhen no sessions are limited by their desired rates or the peernodes.
More formal descriptions of the sink and source nodealgorithms is as follows.
Algorithm 2: (Sink Algorithm)Input: x(t), K, V R. Output: x̂r(t + 1).1. foreach v, v ∈ V R,2. Let x̂r
k(t + 1) = 0, ∀k ∈ Kv .3. repeat
4. xf =Cv−
∑k,k∈Kv,x̂r
k(t+1)6=0
xk(t)
|{k|x̂rj(t+1)=0,k∈Kv}|
5. k = arg minj xj(t), for all j ∈ Kv and x̂rj(t+1) 6= 0;
6. if xk(t) < xf
7. ∆x = α× (xf − xk(t));8. ∆x = min( ∆x, α · xf );9. ∆x = max( ∆x, β · · × xf );
5
10. x̂rk(t + 1) = xk(t) + ∆x;
11. endif12. until|{j|x̂r
j(t + 1) = 0, j ∈ Kv}| = 0 or xk(t) ≥ xf
13. for all k ∈ Kv
14. if x̂rk(t + 1) = 0
15. x̂rk(t + 1) = xf ;
16. endif17. endfor
The sink algorithm includes upper and lower bounds on theadaptation steps (α, β between 0 and 1). An upper boundensures that expected rate does not change too quickly, andlimits overshoot. A lower bound supports faster convergencewhen current rates are close to the target rate.
Algorithm 3: Source Algorithm The source algorithm cal-culates source expected rates in fashion similar to the sinkalgorithm. We use a different notation x̂s
k is used to denotethe expected rate at the source.
Therefore the actual sending rate of a session k ∈ K in timeslot t+1 is the minimum among session k’s desired rate, sinkexpected rate, and source expected rate, defined as
xk(t + 1) = min{x̂sk(t + 1), x̂r
k(t + 1),Mk}. (5)
Intuitively, for flows with low session rates we have xf >xk, meaning that the target rate is higher than the currentsession, giving x̂ > xk. As a result, sessions with low rateshave room to increase if so desired by the application. If asession’s Mk increases beyond the target rate, and there isavailable capacity, it will be allocated more rate.
C. Stability and Convergence
We study the stability and convergence properties of the dis-tributed rate allocation algorithm. In particular, we show that(a) the distributed algorithm has the max-min fair allocationx∗ in the global algorithm as an equilibrium point and (b)the distributed algorithm causes the session rates to converge,regardless of their initial states. Formally,
Proposition 2: Stability. The max-min fair rate vector x∗
calculated by the global algorithm is an equilibrium point forthe distributed algorithm, i.e. if in time slot t,
x(t) = x∗,
then we have thatx(t + 1) = x∗.
The proof for Proposition 2 is included in Appendix B.Proposition 2 essentially says that the set of equilibrium ratesfor the distributed algorithm are the same as those calculatedby the global algorithm. Next, in Proposition 3, we show thatunder the distributed rate allocation algorithm the networkconverges to this equilibrium rate allocation.
Proposition 3: (Convergence) For any initial rate vectorx(0) (at time t = 0), under the distributed rate allocationalgorithm the rate vector x converges to the final rate vectorx∗ computed by the global algorithm in a finite number ofsteps, i.e. there exists time t0 > 0 and for any time slot t > t0,we have that
x(t) = x∗.
We provide a proof of Proposition 3 in Appendix C. The keyinsight is that the leximin ordering (as defined in AppendixA) of the rates can be proven to move toward the equilibriumrate vector.
Combining Proposition 2 and 3 shows directly that the ratevector x∗ is the unique equilibrium point for the distributedalgorithm. And, this rate vector is the max-min rate allocationcomputed by the global algorithm.
Corollary 1: (Capacity Utilization) Except for the initialstate ( t = 0), the maximum aggregate rate at any node v ∈ Vin time slot t ≥ 1 is limited by (1 + α)Cv , i.e.
∑
k∈Kv
xk(t) ≤ (1 + α)Cv.
Proof: This can be directly got from the following twofacts:
(a) For sessions with lower rates than the fair rate, the rateincrease is no more than α · Cv/|Kv|.
(b) For sessions with higher rates than the fair rate, theywill be assigned with fair rate, with leads to the aggregaterate when not counting the increment in (a) equal to the nodecapacity Cv .
In summary, we have shown that there exists a single max-min rate allocation for a network, and that rate allocation iscomputed by both the global algorithm and the distributedrate allocation algorithm. Thus, the distributed rate allocationalgorithm is fair. Further, we have shown that the distributedrate allocation algorithm converges within a finite number ofsteps, so the distributed rate allocation mechanism is stable.
D. Extension to the Asynchronous Case
We extend the proposed distributed rate allocation algorithmfrom synchronous, discrete time asynchronous, continuoustime. Each session can have variable round-trip times, andeach source and sink node can make rate allocation decisionsasynchronously. More specifically we make the followingassumptions:
• Time is continuous, and each node runs the source/sinkalgorithm at each control interval,
• Sessions have varied round-trip delay and informationpropagates from sink to source with latency of one-halfRTT,
• Each source or sink performs its local rate allocationalgorithm asynchronously, and we explore various, fixedcontrol interval;
• The step-size parameters, α and β, for adjusting rateremain fixed and are the same for each source and sink;
The major part of the algorithm remains unchanged. How-ever each node now make its own rate allocation decisionasynchronously based on information which is delayed byround-trip latencies. We will show through simulations thatthe same stability and convergence results still hold in suchasynchronous settings.
6
Session 1, ... , K
Source K
Source 2
Source 1
Sink
Fig. 1. A single sink, multiple source Topology
V. EMPIRICAL STUDIES: THE SYNCHRONIZED CASE
In this section, we illustrate the stability and convergenceproperties of our distributed algorithm, exploring small net-works and large with a variety of desired rates, traffic patterns,and dynamic desired rate changes.
For session rate behavior, we plot session rates for particularflows versus time. However, this visualization quickly becomesineffective for large numbers of flows, as the plots becomeunreadable. To show the algorithms convergence propertiesfor large networks, we define a distance metric between thecurrent rate vector x(t) and the max-min fair equilibrium x∗
(calculated for example by the global algorithm). Experiment-ing with various distance metrics, we find one that works wellis the 2-norm distance, defined as:
D(t) =[ K∑
i=1
(xi(t)− x∗i )2]1/2
.
Therefore in the equilibrium state, we have D = 0.
A. Single Sink, Multiple Source
We first explore the dynamics of our distributed algorithmand its transient behavior with a case study of 5 sources andsingle sink. As shown in Figure 1, there are 5 sources, eachwith a session terminating at a single sink. The five sessionsare assigned rates 0.1, 0.2, 0.3, 0.4 and 0.5, respectively, andthe capacity of each source and sink is 1. We use distributedalgorithm parameters of α = 0.1 and β = 0.1. In this case,the session desired rates and the receiver are the bottlenecks.
For the experiment, all sessions start at t = 0 with initialrates (0, 0, 0, 0, 0), and different desired rates. The trajectoriesof all sessions and the aggregate rate at the sink are depicted inFigure 2. Within 20 time periods, all sessions have convergedto the equilibrium rates. To explore dynamic adaptation, attime t = 50, session 1’s desired rate is increased from 0.1 to1. The distributed algorithm reacts to this change promptly,and the session rates converge to a new equilibrium rateallocation within a short period with each session gets thesame rate 0.2. To explore session termination, we terminatesession 5, 4, 3, 2 at times t = 100, 150, 200, 250, respectively.Again, the simulation results show that the rates of remainingsessions are able to adapt quickly to the changes, filling upthe vacated capacity, and converging to a new equilibrium rateallocation. We list the equilibrium rate allocations in varioustime slots in Table I.
0 50 100 150 200 250 3000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Time
Rat
e Session 1Session 2Session 3Session 4Session 5Aggregate Rate
t=50, Session 1 desired rate changed to 1
t = 100, Session 5 Terminates
t = 150, Session 4 Terminates
t = 200, Session 3 Terminates
t = 250, Session 2 Terminates
Fig. 2. Session rate trajectories for each of five sessions in a scenario withsingle sink and five sources. The five sessions are assigned rates 0.1, 0.2,0.3, 0.4 and 0.5. They terminate in different time slots.
TABLE IEQUILIBRIUM RATES OF FIVE SESSIONS AND THE RATE AT THE SINK
Time 0-50 51-100 101-150 151-200 201-250 251-Session 1 0.1 0.2 0.267 0.5 0.8 1Session 2 0.2 0.2 0.2 0.2 0.2 0Session 3 0.233 0.2 0.267 0.3 0 0Session 4 0.233 0.2 0.267 0 0 0Session 5 0.233 0.2 0 0 0 0Sink Rate 1 1 1 1 1 1
Next, we study the importance of distributed algorithmparameters α and β on convergence. For the same 5-to-1scenario, we plot the distance D of the current rate vectorfrom the equilibrium over time, varying α from 0.05 to 0.2,with fixed β = 0.1. (see Figure 3) Our results show thatlarger α values can enable more rapid convergence. Howeveras we discuss in Corollary 1 that a large α may lead to higherovershoot in some cases. For the same 5-to-1 scenario, weplot the distance D for various values of β (see Figure 4).Our results show that larger β value help speed convergencewhen the system is close to the equilibrium point.
0 10 20 30 40 50 60 700
0.1
0.2
0.3
0.4
Time
Dis
tanc
e
α = 0.05 α = 0.1 α = 0.15 α = 0.2
Fig. 3. The five-source, single-sink scenario: the 2-norm distance betweencurrent rates and equilibrium rates in the cases of different parameter α.
7
0 5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
Time
Dis
tanc
e
β = 0.05 β = 0.1 β = 0.15 β = 0.2
Fig. 4. The five-source, single-sink scenario: the 2-norm distance betweencurrent rates and equilibrium rates in the cases of different parameter β.
1
2
3
456
789
Source 1, C = 1
Source 2, C = 0.6
Source 3, C = 1 Sink 3, C = 1
Sink 2, C = 1
Sink 1, C=0.6
Fig. 5. A topology with three sources and three sinks.
B. Multiple Sink, Multiple Source
We now consider a scenario with three sources and threesinks. Each source and sink participates in three sessionswith connections and node capacities as shown in Figure 5.Note that source 2 and sink 1 have lower capacities thanthe rest of end nodes. All sessions have the same desiredrate of 1. All sessions start at time t = 0 with differentinitial rates (0.02, 0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.16, 0.18).We use distributed algorithm parameters α = 0.1 and β = 0.1.We show the trajectories of each session and the aggregate rateat each source node in Figure 6.
0 5 10 15 20 25 30 35 400
0.5
1
Time
Rat
e
Session 7Session 8Session 9Source 3
0 5 10 15 20 25 30 35 400
0.2
0.4
0.6
Rat
e
Session 4Session 5Session 6Source 2
0 5 10 15 20 25 30 35 400
0.5
1
Rat
e
Session 1Session 2Session 3Source 1
Fig. 6. The three-source, three-sink scenario: the trajectories of each sessionand the aggregate rate at three source nodes.
0 10 20 30 40 50 60 70 800
0.2
0.4
0.6
0.8
1
1.2
Time
Dis
tanc
e
Fig. 7. The 32-node network scenario: network 2-norm distance for 200 testcases
We observe from Figure 6 that the rate of each session aswell as the aggregate rate at each source converges to theequilibrium point within 30 time slots.
C. Larger Networks
While lambda networks are likely to be much smallerthan the internet, the scalability properties of our distributedrate allocation algorithm are still of interest. We evaluatethe convergence properties of the distributed algorithm overa range of network sizes with random traffic patterns asdescribed below. Networks of 32, 128, and 1024 end nodesare used. Each end node has unit capacity. In each scenario,half of the nodes are sources and the other half are sinks. Leteach source participate in 4 sessions with randomly pickedsinks. We generate up to 200 test cases for each of the threescenarios with randomly picked parameters following uniformdistributions:• Step size α: between 0.05 and 0.25;• Step size β: between 0.05 and 0.25;• Desired rates Mk: between 0.1 and 0.5;• Initial rates xk(0): between 0 and Mk.Figure 7, 8, and 9 show our simulation results, plotting
the 2-norm distance between current rates and the equilibriumrates over time of the 50 testing cases for the three scenariosof 32, 128 and 1024 nodes. We see that although the initialdistance varies due to different number of active sessions andinitial state, the distributed algorithm causes the distance todecrease quickly, reaching the equilibrium in no more than 80time slots. Thus our results are promising, illustrating that ourdistributed algorithm is able to converge in a significant rangeof networks and conditions.
VI. EMPIRICAL STUDIES: THE ASYNCHRONOUS CASE
In this section, we study the distributed rate allocationalgorithm in asynchronous scenarios. We consider variousround-trip delay between each source and sink node, andlet each node make rate allocation decisions asynchronously.We conduct ns2 simulations to prove that the stability andconvergence properties of our distributed algorithm still holdunder such fully distributed settings. In all the experiments,the node capacity is set as 500Mbps and normalized to 1. Thecontrol interval at each node and the RTT delay between eachsource and sink are randomly generated. The ns-2 simulationresults are presented in the following subsections.
8
0 10 20 30 40 50 60 70 800
0.5
1
1.5
2
2.5
Time
Dis
tanc
e
Fig. 8. The 128-node network scenario: 2-norm distance for 200 test cases
0 10 20 30 40 50 60 70 800
1
2
3
4
5
6
7
Time
Dis
tanc
e
Fig. 9. The 1024-node network scenario: 2-norm distance for 50 test cases
A. A Multipoint-to-Point Case
We let five sessions start at time t = 0, 8, 16, 24 and 32,respectively. The RTT is generated between 1ms and 100ms(100ms, 75ms, 50ms, 25ms and 1ms); The control interval ofeach node is randomly set between 10ms and 100ms. Figure 10depicts the rate trajectories of each session. We observe thatafter each new session joins, the system converges to a newequilibrium state, in which all sessions get the same share ofthe sink bandwidth.
B. A Four-to-four Case
We now show the rate trajectories and convergence resultswhen there are four sources and four sinks. We setup onesession between each source and sink nodes, thus there are
0 5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Time
Rat
e
100ms75ms50ms25ms1ms
Fig. 10. The trajectories of 5 sessions between five source nodes and onesink node. The 5 sessions start at various times.
0 1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
Time
Rat
e
0 1 2 3 4 5 6 7 8 9 100
0.1
0.2
0.3
0.4
Time
Rat
e
Fig. 11. The trajectories of 16 sessions in a 4-to-4 case. Top: Step sizeβ = 0.2; Bottom: Step size β = 0.05.
16 sessions in total. The sources and sinks have the samecapacity, which is 500Mbps, and normalized to 1. The initialrate of each session is 0. The RTT is randomly generatedwith uniform distribution between 1ms and 100ms. The controlinterval of each node varies between 10ms and 100ms. Stepsize parameters α is equal to 0.1. The trajectories of eachsessions are depicted in Figure 11, with two different valuesof parameter β = 0.2 and 0.05. We see that the rates of all 16sessions converge to 0.25, and a smaller parameter β leads tolonger time for the system to converge.
C. Larger Networks
Network sizes considered are 32, 128, and 1024 end nodes.Each end node has the same capacity, which is normalized to1. In each scenario, half of the nodes are sources and the otherhalf are sinks. We let each source participate in 4 sessions withrandomly picked sinks. We generate 30 test cases for each ofthe three scenarios with randomly picked parameters followinguniform distributions:• Step size α: between 0.05 and 0.15;• Step size β: between 0.05 and 0.15;• RTT: between 1 ms and 100 ms;• Control snterval: between 10 ms and 100 ms.To verify the convergence properties, we still use the 2-
norm distance defined in the previous section. Figure 12, 13and 14 plot the 2-norm distance between current rates andthe equilibrium rates over time of the 30 testing cases for thethree scenarios of 32, 128 and 1024 nodes. To compare withthe results from the synchronous case, in each figure we alsoplot three 2-norm distance trajectories in synchronous casesby setting each time slot to be 1ms, 50ms, and 100ms.
From all three figures we see that the distributed algorithmcauses the 2-norm distance to decrease quickly from initialstates, reaching the equilibrium in no more than 6 seconds. The2-norm distance is also bounded between two synchronouscases with minimum (1ms) RTT and maximum (100ms) RTT.
9
0 1 2 3 4 5 6 70
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Time
Dis
tanc
e
100ms RTT 50ms RTT 1ms RTT
Fig. 12. The 32-node network case: a comparison between the 2-normdistances of 30 test cases in the asynchronous case and the 2-norm distancetrajectories in the synchronous case with various time slot size ( 1ms, 50msand 100ms).
0 1 2 3 4 5 60
0.5
1
1.5
2
2.5
3
3.5
4
Time
Dis
tanc
e
100ms RTT 50ms RTT 1ms RTT
Fig. 13. The 128-node network case: a comparison between the 2-normdistances of 30 test cases in the asynchronous case and the 2-norm distancetrajectories in the synchronous case with various time slot size ( 1ms, 50msand 100ms)
Figure 12 shows the trajectories of all 64 sessions in thecase with 32 nodes.
In conclusion, the results from ns-2 simulations verify thatthe stability and convergence properties of the distributedalgorithm we proposed in the previous section still hold underasynchronous distributed settings. As the asynchronous dis-tributed settings we consider share similar scale and round-triptime variance with to the real lambda-networks, the distributedrate allocation algorithm is proved to be a promising approachfor such lambda-network environments.
VII. RELATED WORK
Delivering end-to-end high-performance communication inhigh bandwidth-delay product networks is a long-standing re-search challenge in wired, wireless and satellite networks [19],[40]. The commonly used transport protocol, standard TCP(TCP Reno) [10], was designed for shared low-bandwidth
0 1 2 3 4 5 60
2
4
6
8
10
12
Time
Dis
tanc
e
100ms RTT 50ms RTT 1ms RTT
Fig. 14. The 1024-node network case: a comparison between the 2-normdistances of 30 test cases in the asynchronous case and the 2-norm distancetrajectories in the synchronous case with various time slot size ( 1ms, 50msand 100ms)
0 2 4 6 8 10 120
0.1
0.2
0.3
0.4
0.5
Time
Rat
e
Fig. 15. The trajectories of 64 sessions in an asynchronous 32-node networkcase
networks. As a result the protocol is inefficient in providingsatisfactory performance in high-speed environments. Theprotocol reacts adversely to the increases in bandwidth anddelay [30], and spends too long to recover from packet lossand reach high bandwidth due to the Additive Increase andMultiplicative Decrease (AIMD) control law [16].
Different approaches have been used to improve standardTCP and achieve a high performance transmission in fastlong-delay networks. First, several approaches preserve theTCP control scheme, but improve the implementation of TCPstack [15], [20], place performance enhancing proxies alonglong-delay links [13], or use parallel connections [25]. Sec-ond, a number of high performance TCP control alternativeshave been proposed to improve TCP performance in shared,packet-switched networks (e.g. High Speed TCP [21], ScalableTCP [36], BIC-TCP [49]). Third, a new set of approachesexclude the loss-based schemes used in TCP, thus exploring
10
the delay-based (e.g. FAST-TCP [33]), and rate-based (e.g.NETBLT [17], RBUDP [26], Tsunami [9], and GTP [48] )data transport protocols.
Rate-based max-min fair bandwidth allocation has beendrawn much research attention [27], [14], [28], [43], [32] inthe context of available bit rate (ABR) service in the ATMnetworks. Such schemes allocate the capacities of switches toactive sessions in a max-min fair manner. They usually assumeinfinite session demands, or explicit knowledge regarding thedesired rate. Instead our work considers the allocation ofendpoint’s capacity, and we assume each endpoint has noknowledge about the desired session rate. Our rate allocationand adaptation scheme discovers each session’s desired ratescontinuously.
In contrast to most of the max-min scheme in ATM net-works which only allocate rates with limited rate adaptation,XCP [35] lets each router make rate feedback based onavailable capacity and queue size. This significantly improvesthe performance over flow transitions. XCP does not conductrate allocation and adaptation at end nodes. Under XCP,sessions with finite desired rate may receive an arbitrarilysmall fraction of its max-min allocation in in a networkedcase with multiple sources and multiple sinks [37].
Multipoint-to-point transmission has been proposed for webtraffic [24] and content delivery network [44]. There are someresearch projects sharing the same idea of endpoint basedmanagement across flows. In [11], a receiver-side integratedcongestion management architecture is proposed, which tar-gets managing traffic across various protocols for real-timetraffic. However detailed rate allocation schemes and fairnessamong flows are not considered. Examples of receiver centricapproaches also include [29] for wireless networks. In nearlyall cases, these studies focus on networks that are slow relativeto the nodes attached to them. In contrast we study the problemof how to allocation the capacity of both sources and sinksamong all sessions in the lambda-networks, which has high-speed low-loss core. Furthermore both efficiency and fairnessare our objectives for bandwidth sharing.
VIII. SUMMARY AND FUTURE WORK
We considered the problem of efficient and fair sharingof source and sink capacities among communication sessionsnetworks with plentiful bandwidth in the core (e.g. lambda-networks). The key challenge in such systems can be formu-lated as a distributed rate allocation and adaptation problem inwhich the desired session communication rates are unknown.First, we presented a global algorithm which computed themax-min optimal rate allocation based on global information,including desired session rates. We then presented a distributedalgorithm which achieves the same end, but without thebenefit of global information or desired session rates. Proofthat the distributed algorithm converges to the same max-min rate allocation as the global algorithm ensures that goodallocations will be achieved. The distributed algorithm is alsosignificantly more flexible because it does not require any“need” information from the applications, rather discoveringit implicitly. Finally, we evaluated the distributed algorithm
empirically, demonstrating its stability and rapid convergencein a range of network scenarios in both synchronous andasynchronous settings.
We have identified the following future work. First, weplan to formally prove the stability and convergence propertiesof distributed algorithm in asynchronous settings, in whicheach sessions may have varied control decision intervals andvariable round-trip delays. Second, we plan to integrate the dis-tributed algorithm into a new version of our Group TransportProtocol [48], and demonstrate the stability and convergenceproperties on various lambda-network testbeds [45]. Third, asthe techniques used in the distributed algorithm may also beapplied to the network with internal bottlenecks, it may beinteresting to see if the stability and convergence properties ofthe distributed algorithm we proposed can still hold in suchgeneral network environments.
ACKNOWLEDGEMENT
This work was supported in part by the National ScienceFoundation under awards NSF Cooperative Agreement ANI-0225642 (OptIPuter), NSF CCR-0331645 (VGrADS), NSFACI-0305390, and NSF Research Infrastructure Grant EIA-0303622. Support from the UCSD Center for NetworkedSystems, BigBangwidth, and Fujitsu is also gratefully ac-knowledged.
REFERENCES
[1] Biomedical informatics research network (birn). http://www.nbirn.net.[2] Bittorrent. http://bitconjurer.org/BitTorrent/.[3] Canarie. http://www.canarie.ca.[4] Griphyn project. http://www.griphyn.org.[5] International virtual data grid laboratory (ivdgl). http://www.ivdgl.org.[6] Kazaa. http://www.kazaa.com.[7] Naregi. http://www.naregi.org.[8] Netherlight. http://www.netherlight.net.[9] Tsunami. Presented at PFLDnet2003, available from
http://steinbeck.ucs.indiana.edu/ mmeiss/papers.html.[10] M. Allman, V. Paxson, and W. Stevens. Rfc 2581: Tcp congestion
control. April 1999.[11] H. Balakrishnan, H. Rahul, and S. Seshan. An integrated congestion
management architecture for internet hosts. Proceedings of ACMSIGCOMM 1999.
[12] D. P. Bertsekas and R. Gallager. Data networks. Prentice-Hall,Englewood-Cliffs, New Jersey, 1992.
[13] J. Border, M. Kojo, J. Griner, and G. Montenegro. Performanceenhancing proxies. RFC 3135, Nov 2000.
[14] A. Charny, D. D. Clark, and R. Jain. Congestion control with explicit rateindication. Proc. IEEE International Conference on Communications(ICC’95), June 1995.
[15] J. Chase, A. Gallatin, and K. Yocum. End system optimizations forhigh-speed TCP. IEEE Communications Magazine, 39(4):68–74, 2001.
[16] D. M. Chiu and R. Jain. Analysis of the increase and decrease algorithmsfor congestion avoidance in computer networks. Computer Networks andISDN Systems, 17(1):1–14, June 1989.
[17] D. D. Clark, M. Lambert, and L. Zhang. Netblt: a high throughputtransport protocol. In Proceedings of ACM SIGCOMM 1987.
[18] T. DeFanti, C. Laat, J. Mambretti, K. Neggers, and B. Arnaud. Trans-light: a global-scale lambdagrid for e-science. Communications of theACM (CACM), 47(11), November 2003.
[19] A. Falk, T. Faber, J. Bannister, A. Chien, R. Grossman, and J. Leigh.Transport protocols for high performance. Communications of the ACM(CACM), 47(11), November 2003.
[20] M. Fisk and W. Feng. Dynamic right-sizing in tcp. In Proc. of the LosAlamos Computer Science Institute Symposium, October 2001.
[21] S. Floyd. Highspeed tcp for large congestion windows. Internet draft,August 2002.
11
[22] S. Floyd, S. Ratnasamy, and S. Shenker. Modifying tcp’scongestion control for high speeds. Available from URLhttp://www.icir.org/floyd/hstcp.html.
[23] T. Greene. Dwdm is the right rx for new york pres-byterian hospital. Network World, 05/16/05, available athttp://www.networkworld.com/news/2005/051605-presbyterian.html.
[24] R. Gupta, M. Chen, S. McCanne, and J. Walrand. Webtp: A receiver-driven web transport protocol. University of California at BerkeleyTechnical Report, 1998.
[25] T. J. Hacker, B. D. Noble, and B. D. Athey. Improving throughputand maintaining fairness using parallel tcp. In proceedings of IEEEINFOCOM 2004, Hongkong, March 2004.
[26] E. He, J. Leigh, O. Yu, and T. DeFanti. Reliable blast udp: predictablehigh performance bulk data transferrbudp. IEEE Cluster Computing,2002, p. 317.
[27] Y. Hou, B. Li, S. Panwar, and H.Tzeng. On network bandwidthallocation policies and feedback control algorithms for packet networks.Computer Networks, vol.34, pp.481-501, 2000.
[28] Y. T. Hou, H. Tzeng, S. S. Panwar, and V. P. Kumar. A generalizedmax-min rate allocation policy and its distributed implementation usingthe abr flow control mechanism. Proceedings of IEEE INFOCOM 1998,pp.1366-1375, San Francisco, CA, 1998.
[29] H.-Y. Hsieh, K.-H. Kim, Y. Zhu, and R. Sivakumar. A receiver-centric transport protocol for mobile hosts with heterogeneous wirelessinterfaces. Proceedings of the 9th annual international conference onMobile computing and networking, 2003: p. 1-15.
[30] B. Jacobson. Tcp extensions for high performance. RFC 1323, May1992.
[31] V. Jacobson. Congestion avoidance and control. Proceedings of ACMSIGCOMM 1988 pp.314-329, August 1988.
[32] R. Jain, S. Kalyanaraman, and R. Viswanathan. The osu scheme forcongestion avoidance in atm networks: Lessons learnt and extensions.Performance Evaluation Journal, Vol. 31, Dec 1997.
[33] C. Jin, D. Wei, and S. Low. Fast tcp: Motivation, architecture,algorithms, and performance. In proceedings of IEEE INFOCOM 2004,Hongkong, March 2004.
[34] P. Karbhari, E. Zegura, and M. Ammar. Multipoint-to-point sessionfairness in the internet. Proceedings of IEEE INFOCOM 2003.
[35] D. Katabi, M. Handley, and C. Rohrs. Internet congestion control forhigh bandwidth delay product network. Proceedings of ACM SIGCOMM2002, Pittsburgh, Aug 2002.
[36] T. Kelly. Scalable tcp: Improving performance in highspeed wide areanetworks, Feb 2003.
[37] S. H. Low, L. L. H. Andrew, and B. P. Wydrowski. Understanding xcp:Equilibrium and fairness. Proc. IEEE Infocom 2005, Miami, FL, March2005.
[38] P. Mehra, A. Zakhor, and C. Vleeschouwer. Receiver-driven bandwidthsharing for tcp. Proceedings of IEEE INFOCOM 2003.
[39] C. Metz. The bright side of dark fiber optics. PC Magazine, availableat http://www.pcmag.com/article2/0,1759,1813381,00.asp.
[40] G. Montenegro, S. Dawkins, M. Kojo, V. Magret, and N. Vaidya. Longthin networks. Jan 2000.
[41] B. Radunovic and J. L. Boudec. A unified framework for max-min andmin-max fairness with applications. Proceedings of 40th Annual AllertonConference on Communication, Control, and Computing, Allerton, IL,October 2002.
[42] B. Radunovic and J.-Y. L. Boudec. Rate performance objectives ofmulti-hop wireless. Proc. IEEE Infocom 2005, Miami, FL, March 2005.
[43] S. H. Rhee and T. Konstantopoulos. Achieving max-min fairness bydecentralization for the abr traffic control in atm networks. IEICE Trans.Commun., Vol. E84-B, No.8, August.
[44] P. Rodriguez and E. Biersack. Dynamic parallel access to replicatedcontent in the internet. IEEE/ACM Transactions on Networking, 2002.10(4): p. 455-465.
[45] L. Smarr, A. A. Chien, T. DeFanti, J. Leigh, and P. Papadopoulos. Theoptiputer. Communications of the ACM (CACM), 47(11), November2003.
[46] N. Taesombut and A. A. Chien. Distributed virtual computer(dvc):Simplifying the development of high performance grid applications. InProceedings of the Workshop on Grids and Advanced Networks (GAN04), April 2004, Chicago, Illinois.
[47] E. Weigle and A. A. Chien. The composite endpoint protocol (cep):Scalable endpoints for terabit flows. Proceedings of CCGrid 2005).
[48] X. Wu and A. Chien. Gtp: Group transport protocol for lambda-grid.Proceedings of CCGrid 2004.
[49] L. Xu, K. Harfoush, and I. Rhee. Binary increase congestion control forfast long-distance networks. In proceedings of IEEE INFOCOM 2004,Hongkong, March 2004.
APPENDIX
A. Proof of Proposition 1
Before proving Proposition 1 we introduce two definitions:leximin mapping and leximin ordering.
Definition 5: (Leximin Mapping). The leximin order map-ping τ : RN → RN is the mapping that sorts vector x innon-decreasing order. We have that
τ(x) = (x(1), ..., x(n)),
where x(1) ≤ x(2) ≤ ... ≤ x(n).We let τm(x) denote the vector of first m elements from
τ(x), which is (x(1), ..., x(m)). We define the leximin reversemapping function φ(·) as
φ(x(j)) = i,
where xi corresponds to x(j) after leximin mapping.In the following, we refer x(i) to the ith element of the
leximin mapping of vector x.Definition 6: (Leximin Ordering) Given that
(x(1), ..., x(n)) and (y(1), ..., y(n)) are the leximin mappingsof x and y, we define the lexicographic ordering of xand y as x ÂL y, if and only if (∃i > 0)x(i) > y(i) and(∀0 < j < i)x(j) = y(j). We also define x ºL y, if and onlyif x ÂL y or x = y; and x ≺L y, if and only if y ÂL x.
We let ≡Ln denote a relation between two vectors x and
y, such that x ≡Ln y if and only if τn(x) = τn(y), and
xφ(x(i)) = yφ(x(i)), i = 1..n.It is shown in [41] that if a max-min fair vector exists, then
it is the unique leximin maximal vector. Thus the existenceof a max-min fair vector implies the uniqueness of a leximinmaximum. Now we provide a proof to Proposition 1.
Proof: (Proposition 1.) As in the global algorithm(Algorithm 1) at least one undetermined session is assignedwith a rate in each iteration, the total number of iterationsrequired for Algorithm 1 is then no more than K iterations,where K is the total number of active sessions. This provesthe correctness of the first statement.
The Proposition 3 in [41] shows that if a max-min fair vectorexists, it is the unique leximin maximal vector. As it is straightforward that the calculated rate vector x∗ is feasible, we areto prove by contradiction that if there is another feasible ratevector y, and y >L x∗, then y is not feasible.
Suppose there exists m ∈ [1, .., K] , such that y(m) > x∗(m)
and τm−1(x∗) = τm−1(y).We firstly show by induction on p = 1..m− 1 that
x∗φ(x∗(p))= yφ(x∗(p))
.
(1) When p = 1, let i = x∗(1)1. As y(1) = x∗(1), for allk = 1..K, we have
xi = x∗(1) ≤ yk.
Therefore either x∗i = yi or x∗i < yi.
12
We consider the case when x∗i < yi. From global algorithmwe know that x∗i = min{Mi, b(v)}, where v is the corre-sponding node that has the smallest b(v) in the ith iteration.
If x∗i = Mi then yi > Mi, which violates the feasibility.Now we consider x∗i = b(v). The global algorithm impliesthat for any j, 0 < j < i, x∗i ≥ x∗j . Then we have
x∗i = b(v) ≥ Cv/|Kv|.And as for any j (1 ≤ j ≤ K), yj > x′∗1 , then for the samenode v we have that
∑
j,j∈Kv
yj ≥ y(1) · |Kv| > x∗(1) · |Kv| ≥ Cv,
which also violates the feasibility condition.As it is impossible to have x∗i < yi for a feasible y, we
obtain that x∗i = yi.(2) Suppose that τp(x) = τp(y), and for all j′ ∈ [1, p− 1],
we havex∗φ(x∗
(j′))= yφ(x∗
(j′)). (6)
We show that x∗φ(x∗(p))= yφ(x∗(p))
.
Let i = φ(x∗(p)). We show by contradiction that both Case(2.a) yi < xi and Case (2.b) yi > xi are not possible.
Case (2.a): If yi < xi then yi < y(p). This implies thatthere exits q > 0 and q 6= p, s.t. i = φ(y(q)). Then we haveq < p. However from Eq. 7 we obtain that yφ(x∗(1))
..yφ(x∗(p−1))
are p− 1 unique ones to y(q) and are all less than y(p). As aconsequence there are p elements that are less than y(p), whichis impossible according to the definition of lexmin mapping.
Case (2.b): If yi > xi, then from the global algorithm weknow that
x∗i = min{Mi, b(v)},where v is the corresponding node that has the smallest b(v)in the ith iteration. If x∗i = Mk, then we have yi > Mi whichviolates the feasibility condition. If x∗i = b(v), then we have
∑
j,j∈Kv
y(j) =∑
j,j∈Kv,x∗(j) 6=0
y(j) +∑
j,j∈Kv,x∗(j)=0
y(j)
≥( ∑
j,j∈Kv,x∗(j) 6=0
y(j)
)+y(j) · |{j|x∗(j) = 0, j ∈ Kv}|
>( ∑
j,j∈Kv,x∗(j) 6=0
x∗(j))+x∗p · |{j|x∗(j) = 0, j ∈ Kv}| ≥ C, (7)
which also violates the feasibility condition.By induction from (1) and (2), for p = 1..m− 1, we have
x∗φ(x∗(p))= yφ(x∗(p))
.
It can be proved by applying similar arguments as abovethat if y(m) > x∗(m) then y is not feasible. This proves thesecond statement.
The third statement directly follows from the Proposition 1in [41] (If a max-min fair vector exists on a set, then it isunique).
B. Proof of Proposition 2
Proof: We first to show that if x(t) is feasible, then forany session k ∈ [1, ..K], we have
x̂sk(t + 1) ≥ xk(t), and x̂r
k(t + 1) ≥ xk(t). (8)
Recall that in the distributed algorithm the expected rate x̂k
of session k depends on the target rate xf and its current ratexk(t). And only when xf is less than the current session ratexk(t), the expected rate will be set to xf , thus smaller thanthe current rate:
xk(t + 1) = xf < xk(t).
Eq. 4 implies that this only happens when the remaining capac-ity is less than the aggregate current rate of all undeterminedsessions. In this case we have
∑xk(t) > C.
However this also implies that the rate vector x in time slot tis not feasible. Therefore given any feasible rate vector x intime slot t, the expected rates of each session k in time slott + 1 satisfies Eq. 8.
Recall that the rate of each session k in time slot t isdetermined by
xk(t) = min{x̂sk(t), x̂r
k(t),Mk}.We consider the following two separate cases: (1) xk(t) = Mk,(2) xk(t) < Mk and xk(t) = x̂s
k(t) or xk(t) = x̂rk(t).
Case (1): xk(t) = Mk. We can get by Eq. 8 that
xk(t + 1) = min{x̂rk(t + 1), x̂s
k(t + 1)} ≥ xk(t).
As it is always true that xk(t + 1) ≤ Mk = xk(t), it directlyfollows that xk(t + 1) = xk(t).
Case (2): xk(t) < Mk and xk(t) = x̂sk(t) or xk(t) = x̂r
k(t).This implies that session k is constrained by its source/sinknode but not its desired rate. Let v be the corresponding nodeon which xk(t) equals to its expected rate in time slot t.This also implies that xk(t) = x∗k = xf . We notice thefollowing facts from the global algorithm: (a) As session kis not restricted by its desired rate, the aggregate rate at nodev equals to Cv; (b) both x∗k and xk(t) equal to the highestsession rate among all sessions at node v. Then from thecalculation of xf (Eq. 4) in the distributed algorithm we canobtain in time slot t + 1:
xf =Cv −∑
i∈Kv,x̂i(t+1) 6=0 xi(t)
|{j|x̂j(t + 1) = 0, j ∈ Kv}|
=Cv −∑
i∈Kv,xi(t)<xk(t) xi(t)
|{j|xj(t) ≥ xk(t), j ∈ Kv}|
=Cv −∑
i∈Kv,x∗i<x∗
kx∗i
|{j|x∗j ≥ x∗k, j ∈ Kv}| = x∗k.
Thus we havex̂k(t + 1) = xk(t),
and xk(t + 1) = xk(t).Combining Cases (1) and (2), we have shown that the rate
vector calculated by the global algorithm is an equilibrium
13
point for the distributed algorithm. By Proposition 1, theequilibrium point is max-min fair and unique.
C. Proof of Proposition 3
The proof of Proposition 3 uses the notion of leximinordering as defined in Appendix A. We show that the ratevector x converges to x∗ in lexinmin order, meaning that (1)among unconverged sessions the one with smallest equilibriumrate will converge next, and (2) once a set of sessions withsmallest equilibrium rates converge, they will retain such ratesafterwards. Intuitively this is due to the fact that the distributedalgorithm gives higher priority to consider the sessions withsmall rates.
Before proving Proposition 3, we state several preliminarylemmas, and give an outline the proof for each of them.
Lemma 1: Let x∗ be the result of the global algorithm. Letx(t) be the rate vector of the distributed algorithm in time slott. If there exists t1 ≥ 0 and there exists n ∈ [1..K], s.t.
x(t1) ≡Ln x∗, (9)
then for any t′ > t1, it holds that
x(t′) ≡Ln x∗. (10)
Proof: Given Eq. 9, we are to prove the lemma byshowing that the following two statements are always true.
If Eq. 9 is true for time slot t, then in time slot t + 1 wehave that
(a) For any j ∈ [1..n], let i = φ(x(j)). Then it holds that
xi(t + 1) = xi(t);
and (b) For any j ∈ [n + 1..K], it is true that
x(j)(t + 1) ≥ x(n)(t).
We prove (a) by induction on j = 1..n.(1) When j = 1, let i = φ(x(1)). Then we have
xi(t) = x∗i = x(1).
We consider two cases: (1a): xi(t) = Mi and (1b): xk(t) <Mk and xk(t) = x̂s
k(t) or xk(t) = x̂rk(t). Similar arguments
as used in the proof of Proposition 2 can be applied, and itcan be proved that
xi(t + 1) = xi(t) = x∗(t).
(2) Suppose (a) is true for 1..j − 1 (j > 0). We need toshow that the same is true for j, i.e. if i = φ(x(j)) then wehave
xi(t + 1) = xi(t).
Notice the fact that the calculation of x̂ri (t + 1) and x̂s
i (t + 1)only depends on the source/sink capacity of session i andsessions with lower rate than xi(t) at the same source orsink node. This is also true when calculating x∗i in the globalalgorithm. Notice the similarity between b(v) and xf whenx(t) ≡L
j−1 x∗, and by Eq. 5 and 2 it can be proved that
xi(t + 1) = xi(t) = x∗i .
Now we show the correctness of (b). Suppose that thereexists j ∈ [n + 1, K], and let i = φ(x(j)), s.t.
xi(t + 1) < x(n)(t) = x∗(n). (11)
As by the definition of leximin ordering, xi(t) ≥ x(n)(t), wethen have
xi(t + 1) < xi(t). (12)
As stated before, Eq. 12 is only valid when xi(t) > xf , wherexf is the target rate at the source or sink of session i whencalculating its expected rate. However from the fact that
x(t) ≡Lj−1 x∗,
we have
xf =C −∑
k,k∈Kv,x̂rk6=0 xk(t)
|{k|x̂j = 0, k ∈ Kv}| ≥Cv −∑n−1
i=1 x∗(i)K − n + 1
≥ x∗(n).
Therefore xi(t + 1) > x∗(n), which contradicts with Eq. 11.
Lemma 1 implies that if a subset of rates which are thelowest ones of x converges to the subset of lowest rates ofx∗, then these rates will stay unchanged afterward. The nextlemma is to show that any rate vector that is lexicographicallyless than x∗ will monotonically increase in the leximin orderin the next time slot.
Lemma 2: Let x∗ be the result of the global algorithm. Letx(t) be the rate vector of the distributed algorithm in time slott. If
x(t) ≺L x∗,
then in time slot t + 1 we have that
x(t) ≺L x(t + 1).Proof: As x(t) ≺L x∗, by definition there exists n ∈
(0,K), s.t. x(t) ≡Ln−1 x∗ and (1) x(n)(t) < x∗(n) or (2)
x(n)(t) = x∗(n), and xτ(x(n)) < x∗τ(x(n)).
For case (1), by applying Lemma 1 we obtain that in timeslot t + 1,
x(t + 1) ≡Ln−1 x∗.
Then it is sufficient to show that in time slot t + 1,
x(n)(t + 1) > x(n)(t). (13)
and for any l ∈ (n + 1,K]
x(l)(t + 1) ≥ x(n)(t). (14)
Let i = φ(x(n)(t)). As is shown in the proof of Lemma 2that only when the remaining capacity over the number ofundetermined sessions (xf ) is less than the current minimumrate among undetermined sessions, we have x̂k(t+1) < xk(t).However as
xi(t) = x(n)(t) < x∗(n) ≤ x∗i ≤ xf ,
then we have that Eq. 13 holds.For any l ∈ (n + 1,K], similarly as in case (1), when
x(l)(t) ≥ x∗x(n), we have
x(l)(t + 1) ≥ x∗x(n)> x(n)(t);
14
and when x(l)(t) < x∗x(n), we have
x(l)(t + 1) > x(l)(t) ≥ x(n)(t).
Therefore Eq. 14 is valid.Case (2) can be proved by using similar arguments as in
the proof of case (1).
Lemma 3: Let x∗ be the result of the global algorithm. Letx(t) be the rate vector of the distributed algorithm in timeslot t. If in time slot t1, x(t1) ≡L
n x∗, where 0 ≤ n < K andx(n+1)(t1) 6= x∗(n+1), then there exits t2 > t1, s.t. in any timeslot t > t2 we have that
x(t) ≡Ln+1 x∗ (15)
Proof: Given that in time slot t1, x(t1) ≡Ln x∗, we
consider the following two situations: (1) x(t1) ºL x∗, and(2) x(t1) ≺L x∗.
Case (1): The relations x(t1) ≡Ln x∗ and x(t1) ºL x∗
imply that for any j ∈ [n + 1, K], it holds that
x(j)(t1) ≥ x∗(n).
Therefore it can be easily proved that for any j ∈ [n + 1,K],let i = φ(x(j)), then we have
xi(t + 1) ≥ x∗(n).
We can also show by using similar technique as in the proofof Lemma 1 that when i′ = φ(x(n+1)), then we have
xi′(t + 1) = x∗n+1.
Together with the fact that x(t1 + 1) ≡Ln x∗ (by Lemma 1),
we havex(n+1)(t1 + 1) = x∗(n+1),
and Eq. 15 is valid.Case (2): By applying Lemma 2 we obtain that
x(t1) ≺L x(t1 + 1).
Also from the fact x(t1+1) ≡Ln x∗ and the smallest increment
in the distributed algorithm is α ·β ·Cv/|Kv|. Therefore withinfinite time slots, i.e. there exits time t2 > 0, in time slot t2there is no unconverged session with rate lower than x∗(n+1).We then have that
x(t2) ºL x∗.
This becomes the same case as case (1).To conclude, in both cases (1) and (2), Eq. 15 becomes valid
within finite steps. Lemma 1 ensures that Eq. 15 is alwaysvalid afterwards.
Lemma 3 shows that if the smallest n ∈ [0, .., K − 1]elements in the rate vector converges, then the (n + 1)thelement will converge to its corresponding equilibrium ratewithin finite time slots. Based on these lemmas we now outlinethe proof for Proposition 3.
Proof: Proposition 3 can be proved by applying lemma 3through induction on n from 0 to K-1, i.e. we show thefollowing:
(1) It holds that there exits t1 ≥ 0 and for any t ≥ t1we have that x(t) ≡L
1 x∗. This can be directly obtained fromLemma 3 by setting n = 0.
(2) Suppose there exits tn−1 ≥ 0 and for any t ≥ tn−1 wehave that x(t) ≡L
n−1 x∗. Then there exits tn ≥ tn−1 and forany t ≥ tn we have that x(t) ≡L
n x∗. This is true by applyingLemma 3 and Lemma 1.
Combining (1) and (2), we conclude that there exits tK ≥ 0and in any time slot t ≥ tK we have that
x(t) ≡LK x∗.
This proves the proposition.
