Multipath Routing From a Traffic EngineeringPerspective: How Beneficial is It?

Xuan Liu1, Sudhir Mohanraj1, Michał Pioro2, Deep Medhi11University of Missouri–Kansas City, USA; 2Warsaw Univ. of Technology, Poland & Lund University, Sweden

xuan.liu@mail.umkc.edu, smcp6@mail.umkc.edu, mpp@tele.pw.edu.pl, dmedhi@umkc.edu

Abstract—Multipath routing gives traffic demands an opportu-nity to use multiple paths through a network. In a single-demandsituation, its benefits are easy to see. In a multi-commoditycase, when potentially all node-pairs (demands) generate traffic,they compete for the same network resources. In this work,we consider multipath routing in communication networks in amulti-commodity setting from a traffic engineering perspective.Based on a result from linear programming, we show that at anoptimal solution, the number of demands that can have multiplepaths with nonzero flows is of the order of the number of networklinks for three commonly used traffic engineering objectives. Weintroduce a multipath measure (MPM) and show that undercertain traffic conditions and topological structures, the MPMis zero or close to zero; i.e., multipath routing provides little orlimited gain compared to single-path routing. For the all-pairtraffic case, multipath routing is observed to be advantageousfor small networks. When the number of nodes is about 25 orhigher and all node pairs have traffic, this advantage drops asthe number of nodes in a network increases. For the fat-tree datacenter topology, the benefit of multipath routing also drops asthe number of pods increases. Our findings are somewhat againsta common belief (expressed by the term “load sharing”) thatmultipath routing is significantly better in effective distributionof traffic over the network resources.

I. INTRODUCTION

Multipath routing is a topic that has been studied extensivelyfor a long time. It is widely believed that multipath routing isadvantageous, using more than one way to send traffic froman origin to a destination.

Despite the extensive literature on multipath routing, wenoted that very few authors addressed multipath routing com-prehensively from a traffic engineering (TE) perspective. ByTE, we mean that a network goal is to be optimized fortraffic in a network. For communication networks, commonlyused network goals for TE are minimum cost routing (MCR),load balancing (LB) (also known as congestion minimization),and minimum average delay (AD); these will be elaboratedlater on. In general, MCR is often used for circuit-basednetworks, while LB and AD are commonly used for packet-based networks. The use of these objectives for communi-cation networks goes back to the early days of networkingresearch. MCR for circuit-based networks was discussed in[21], [26]; for packet-based networks, LB was discussed in[25] and AD in [6]. More recently, both LB and AD objectiveshave been used in TE of IP networks by backbone Internet

This work has been partially supported by US National Science Foundationunder Grant Nos. CNS-0916505 and CNS-1217736, and by National ScienceCentre of Poland under Grant No. 2011/01/B/ST7/02967.

service providers [5], [15]. Despite the early work on TE ofcommunication networks, the benefit of multipath routing doesnot appear to be fully addressed nor understood.

Our focus in this paper is on multipath routing for wiredcommunication networks such as an ISP’s backbone network.We consider three objectives, MCR, LB, and AD to understandthe benefit of multipath routing and to present a unifiedframework. Our work stems from a little known result that wecall the D+L property for the MCR objective; in this work,we extend this result to the LB and AD objectives. This allowsus to understand and assess multipath routing from a newperspective for any of the TE objectives. Roughly speaking, theD+L property states that when considering multipath routingoptimization problems, the number of paths that are actuallyutilized is limited by the number of demand pairs (D) plusthe number of links (L). We elaborate on this observation byanalyzing representative network topologies and traffic condi-tions. An important observation is that when all node pairs inthe network have traffic and the system is feasible, multipathrouting tends to perform closer to single-path routing when thenumber of nodes grows, irrespective of the objectives. This isnot meant to suggest that paths should not be changed duringtraffic/capacity fluctuations — they should and the path setsshould be optimized explicitly/implicitly. Our claim is that atany given state/instant, single-path routing is often sufficientif the network is large.

The scope of the paper is as follows. In Section II, wediscuss the TE models, present the D + L property (withproof), and introduce the multipath measure (MPM). In Sec-tion III, we present studies and comparison for a numberof topological structures and network sizes using a varietyof traffic distributions when all node pairs have traffic. InSection IV, we present results for the special case of thefat-tree data center topology. Related work is discussed inSection V. We conclude with a discussion in Section VI.

II. TRAFFIC ENGINEERING MODELS

In order to discuss the three TE objectives, MCR, LB, andAD, we start with the basics that are common to them.

A. Basics

Consider an undirected communication network with Nnodes. A set of demands, i.e., node pairs with offered trafficwill be denoted by D, with traffic hd, d ∈ D. The total externaltraffic offered to the network is then given by H =

∑d∈D hd.

The cardinality of D is denoted by D: D = |D|. If every

(undirected) pair of nodes has traffic, then D = N(N − 1)/2.In general, we assume that there is traffic between everypair of nodes, unless stated otherwise. The network has aset of links L, and the capacity of link ` is denoted by c`(c` > 0, ` ∈ L). In a practical network, multiple parallel linksmay exist between two directly connected nodes; we assumehere that without loss of generality (w.l.o.g), at most one linkexists between any two nodes.

We assume that every demand has at least two paths throughthe network (not necessarily link or node disjoint) allowingus to study multipath routing. For each demand d, the set ofpaths will be denoted by Pd; the average number of pathsfor each demand is denoted by P . We use link-path indicatorδdp`, which is set to 1 if path p for demand d uses link `, andto 0 otherwise. All three models will be described using thelink-path multicommodity flow formulation.

For demand d ∈ D, the non-negative flow variable on pathp ∈ Pd is denoted by xdp. The vector of all flows is defined asx = (xdp : d ∈ D, p ∈ Pd) where x ≥ 0. In the presence ofmultiple paths for each demand, xdp then tells us how muchtraffic might flow on a particular path p. We are especiallyinterested in this for the three TE objectives. For notationalconvenience, we will also use the dependent variable y` todenote the link load, i.e., flow of traffic on link ` that isgiven by y` =

∑d∈D

∑p∈Pd

δdp`xdp, ` ∈ L. For the MCRobjective, we also use the unit cost of a flow on path p ∈ Pd,which will be denoted by ξdp, d ∈ D, p ∈ Pd.

Occasionally, we will need to refer to a specific demand toconsider traffic variation from the rest of the demands. In sucha case, we identify demand d between nodes i and j (i < j)by its end points as i:j, i.e., d ≡ 〈i:j〉. Deviating from theusual notation, its traffic will be denoted by hi:j . Finally, anoptimal solution is identified with an ∗.

B. Minimum Cost Routing (MCR)

The goal of MCR is to minimize the routing cost oftraffic flows on paths. Its link-path multicommodity flow linearprogramming (LP) formulation [21] is shown below:

min{x≥0}

F1(x) =∑d∈D

∑p∈Pd

ξdp xdp (1a)∑p∈Pd

xdp = hd, d ∈ D (1b)∑d∈D

∑p∈Pd

δdp` xdp ≤ c`, ` ∈ L. (1c)

The above formulation has two sets of constraints. The firstset, consisting of D equations (“demand constraints”), assuresthat the traffic is carried, i.e., traffic hd for each demand dis satisfied by the flow variables associated with its set ofpaths. The second set, consisting of L inequalities (“capacityconstraints”), ensures that the traffic flow on link ` does notexceed its capacity c`.

The question of interest to us is: how many flow variablesare necessarily positive at optimality? This question wasanswered in [18, Chapter 4]; the proposition with the proofis reproduced below.

Proposition 1: [18] (D+L property) If the linear program(1) is feasible, then there exists an optimal vertex solution thathas at most D + L positive flow variables.

Proof: The proposition follows from the basic charac-terization of the vertex solutions of linear programs in thestandard form [12]. Introducing non-negative slack variabless = (s` : ` ∈ L), the feasible set (polyhedron) of the standardform of (1) is specified by x ≥ 0, s ≥ 0, and D+L equations

∑p∈Pd

xdp = hd, d ∈ D (2a)∑d∈D

∑p∈Pd

δdp` xdp + s` = c`, ` ∈ L. (2b)

Each vertex solution of (2), i.e., each basic feasible solutionhas at most D+L positive variables out of P ·D+L variablesx and s. This means that any optimal (with respect to (1a))vertex solution has at most D + L positive flow variables.

We consider the above result to be little known since this israrely mentioned in the multipath routing literature despite anumeric illustration of the above result in [1]. There are threeimportant consequences of the D + L property.

Corollary 2: If the linear program (1) is feasible, then anoptimal vertex solution corresponds to at most L demands withmore than one path with positive flows.

Proof: Since by Proposition (1), an optimal vertex solu-tion has at most D+L nonzero flow variables and there are atotal of D demands, at least one path for each demand mustcarry the (nonzero) traffic for the demand. This then leaves uswith at most D + L − D = L demands that can have morethan one path with nonzero flows at optimality.

The above corollary implies that in a network topologywhere D � L, a vast majority of demands is routed on singlepaths. For example, consider when every node pair has traffic,i.e., D = N(N −1)/2. Assume that L = γN (this means that2γ is an average vertex degree). Then, setting γ = 3 (averagevertex degree equal to 6) we have L = 3N and, thus, at most3N out of N(N − 1)/2 demands, i.e., 6

N−1 , are realized onmore than one path. For N = 101, this quantity is exactly 6%.

Corollary 3: If no link is saturated at an optimal vertexsolution x∗ of linear program (1), then each demand uses onlyone positive flow, yielding a single-path optimal solution.

Proof: The above assumption means that s∗ > 0. Thus,L out of D + L nonzero variables are all slack variables s∗.That is, there are exactly D positive flows in x∗, i.e., one perdemand.

Simply put, Corollary 3 says that in lightly loaded networks,single-path routing is optimal.

Corollary 4: The D+L property and Corollaries 2 and 3hold for the whole family of linear programming problems invariables x ≥ 0 and the feasibility set defined by (1b)-(1c).

Proof: This is true because the D + L property holdsfor all vertex solutions of the feasibility set, and hence, forall optimal vertex solutions for an arbitrary linear objectivefunction in variables x.

1

3

2

Capaci

ty = 1

Capacity = 1

Capacity = 1

Traffic = 0.5

Traffic = 0.4Traffic =

1Fig. 1: 3-node example

Corollary 4 implies that the D+L property also applies to

max F2(x) =∑`∈L

(c` −∑d∈D

∑p∈Pd

δdp`xdp), (3)

i.e., maximizing the total unused link capacity that is left afterthe demand allocation.

Consider now a 3-node network (Fig. 1) to illustrate theimplications of the above results; detailed results for largertopologies will be presented later on. In this example, we labeldemands as 1 ≡ 〈1:2〉 (i.e., between 1 and 2), 2 ≡ 〈1:3〉,3 ≡ 〈2:3〉. Similarly, the links are labeled as 1 for link 1-2, 2 for link 1-3, and 3 for link 2-3. Each demand has twopossible paths: one direct and one via the third transit node.Thus, for demand 1 (i.e., between nodes 1 and 2), the flowvariables for two paths are x11 and x12 where the first is onthe direct link 1-2 while the second one takes the path 1-3-2 and similarly, for the other two demands. Given the trafficand the capacity, as listed in Fig. 1, it is clear that only thedirect path will be optimal if the direct path is cheaper thanthe alternate (i.e., ξd1 < ξd2, for d ∈ D). That is, the optimalsolution is x∗11 = 0.5, x∗21 = 1, x∗31 = 0.4, while the othervariables will be zero. Thus, in this case, each demand usedonly a single path at optimality, while multiple paths wereavailable. Obviously, if the traffic volume for demand 2 isincreased to 1.1, then 0.1 unit of traffic would need to takethe second path, thereby resulting in invocation of multipathrouting for this demand. Certainly, this still meets the D + Lproperty in that at most D+L flow variables are nonzero; inthis case, D + L = 6.

C. Load Balancing (LB)

For the LB problem, the goal is to minimize the utilizationof the most utilized link, i.e., to minimize congestion, whichwas originally suggested by Wozencraft [25]. This problemcan be formulated as the following LP problem:

min{x≥0, r}

F3(r) = r (4a)∑p∈Pd

xdp = hd, d ∈ D (4b)∑d∈D

∑p∈Pd

δdp` xdp ≤ c`r, ` ∈ L (4c)

The capacity constraint (4c) can alternatively be written asr ≥

∑d∈D

∑p∈Pd

δdp` xdp

c`so that objective (4a) minimizes the

maximum load-to-capacity ratio over all links, as required.Note that we do not need to assume that r ≤ 1. In effect,there are no capacity constraints and thus, the LB problem (4)

is always feasible as a linear program. Obviously, if r∗ > 1,it means that the network does not have sufficient capacity toserve the traffic load. Note also that, for given c`, ` ∈ L, theoptimal solution paths for formulation (4) obeys traffic scaling.To see this, consider that hd is replaced by βhd, where β > 0.Then, the traffic flows on the paths are scaled by β and theoptimal objective value changes from r∗ to βr∗.

Compared to MCR, the above LP formulation also has D+L constraints but P ·D+ 1 variables. The additional variabler must be greater than 0 since r could be equal to zero onlyif hd = 0, d ∈ D, i.e., if there was no traffic in the network.Thus, we now extend the D+L property for the MCR problemto the D + L− 1 property for the LB objective:

Proposition 5: (D + L − 1 property) For linear program(4), there exists an optimal vertex solution that has at mostD + L− 1 positive flow variables.

The proof makes use of inequality r∗ > 0 and otherwiseis analogous to the proof of Proposition 1. Consequently, forthe LB objective we arrive at the following counterpart ofCorollary 2.

Corollary 6: For linear program (4), an optimal vertexsolution corresponds to at most L − 1 demands with morethan one path with positive flows.

Consider again the three-node example of Fig. 1. This time,we use the load balancing objective. We find that the optimalr∗ = 0.75, and there are multiple optimal flow solutions givenby x∗11 = 0.5 − ζ, x∗12 = ζ, x∗21 = 0.75 − ζ, x∗22 = 0.25 +ζ, x∗31 = 0.4 with 0 ≤ ζ ≤ 0.05. When ζ = 0, 4 flow variablesare nonzero; when 0 < ζ ≤ 0.05, 5 flow variables are nonzero.We can see that demand 3 always uses a single-path for itstraffic at optimality. Reflecting on Proposition 5, we knowthat at most the D+L− 1 = 3 + 3− 1 = 5 flow variables arenonzero at optimality for (4). Indeed, at most 5 flow variablesare nonzero at optimality for this three-node example. Moreinterestingly, because of Proposition 5, we find that for a 3-node network, for the load balancing objective, at least onedemand (out of three demands) must always use a single path,because of the condition of at most 5 nonzero flow variables.In other words, it is not possible for one of the three demandsto use multiple paths at optimality for the LB objective, nomatter what the traffic volume or capacity in the network is.

D. Minimum Average Delay (AD)

The AD optimization problem [6] is as follows:

min{x≥0, y≥0}

F4(y) =1

H

∑`∈L

y`c` − y`

(5a)∑p∈Pd

xdp = hd, d ∈ D (5b)∑d∈D

∑p∈Pd

δdp` xdp = y`, ` ∈ L (5c)

y` ≤ η c`, ` ∈ L. (5d)

The objective function (5a) expresses the average delay expe-rienced by the traffic. Constraints (5c) define the link loadsy = (y` : ` ∈ L), while constraints (5d) assure that the loadof each link is kept below the link capacity multiplied by the

given constant η = GG+1 . Note that G > 0 is the assumed

maximum delay allowable on a link since ηc`c`−ηc` = G.

The above problem is non-linear since the functionf(y; c) = y

c−y is non-linear. In f(y; c), c is a fixed parameter(link capacity) and y, 0 ≤ y < c, is the argument (link load).More precisely, (5) is a minimization problem with a convexcost function (which, in turn, is a sum of one-argument convexfunctions) and linear constraints. As such, it can be arbitrarilyclosely approximated by a linear program, using a piecewiselinear approximation (PLA) of the functions f(y`; c`) for each` [18]. In order to obtain such an approximation, we considerfunction g(z) = z

1−z , 0 ≤ z < 1. This function is convex,strictly increasing in its domain, and g(0) = 0. Let

g(z) = max {akz − bk : k ∈ K} (6)

be a PLA of g(z). In (6), K = {1, 2, . . . ,K}, and ak, bk aregiven coefficients of the K consecutive linear pieces (segmentsof the graph of g(z)). Then, for each ` ∈ L, the functionf(y`; c`) = g(y`c` ), ` ∈ L, is the corresponding PLA ofthe function f(y`; c`). Note that g(z) is a convex functionspecified for all z ∈ R, and so are all functions f(y`; c`).The properties of g(z) make it possible to assume w.l.o.g.0 < a1 < a2 < . . . < aK , b1 = 0 < b2 < . . . < bK , andg(η) = g(η) = G, i.e., aK−1 η − bK−1 = aKη − bK = G. Inthe numerical studies reported in this paper, we used η = 19

20 ,G = 19, and the following PLA of g(z) with K = 6 pieces:

max { 3z2 ,9z2 −1, 15z−8, 50z−36, 200z−171, 4000z−3781}.

The explicit form of the PLA function is as follows:

g(z) =

32 z for 0 ≤ z < 1

392 z − 1, for 1

3 ≤ z <23

15 z − 8, for 23 ≤ z <

45

50 z − 36, for 45 ≤ z <

910

200 z − 171, for 910 ≤ z <

1920

4000 z − 3781, for z ≥ 1920 .

(7)

As required, the equality g(η) = g(η) = 19 holds. Observethat approximation (7) is different (and more accurate) than thewell known PLA studied in [5]. Now we can write down an LPapproximation for problem (5) with objective function F4(y)substituted by F5(y) =

∑`∈L f(y`; c`). Introducing additional

continuous variables z = (z` : ` ∈ L), the formulation of theapproximation is as follows:

min{x≥0y≥0z≥0

} F5(z) =1

H

∑`∈L

z` (8a)

∑p∈Pd

xdp = hd, d ∈ D (8b)∑d∈D

∑p∈Pd

δdp` xdp = y`, ` ∈ L (8c)

z` ≥ aky`c`− bk, ` ∈ L, k ∈ K. (8d)

Constraints (8d), together with the minimization-type objective(8a), assure that z∗` = f(y∗` , c`) for each `. In (8), constraints(5d) are not present so that (8) is always feasible. When

the original problem (5) is feasible, these constraints areredundant. If we assume that the slope aK of the Kth linearpiece in (6) (i.e., the piece that approximates the function g(z)in the interval [η, 1]) is sufficiently steep, then the objectivefunction heavily penalizes the cases with y∗` > η c`.

Clearly, any feasible vertex solution of the linear program(8) has at most D + 2L positive path flows. This is becausefor each link ` ∈ L at most two constraints (8d) can be activeat optimality. On the other hand, for (8) too, the tighter D+Lproperty holds.

Proposition 7: For linear program (8), there exists anoptimal vertex solution that has at most D + L positive flowvariables.

Proof: Suppose that x∗ = (x∗dp : d ∈ D, p ∈ Pd), y∗ =(y∗` : ` ∈ L), z∗ = (z∗` : ` ∈ L) is an optimal solution for (8).Knowing y∗, consider the following system in x(≥ 0):∑

p∈Pd

xdp = hd, d ∈ D (9a)∑d∈D

∑p∈Pd

δdp` xdp = y∗` , ` ∈ L. (9b)

The polytope given by (9) has D + L equations; thus, any ofits vertices x has at most D + L positive flow solutions; thisfollows from the same argument as in Proposition 1.

We claim that x is also optimal for (8). This is easy to seesince there are no changes to y∗ in (9); this, in turn, meansthat there are no changes to z∗ due to active constraints atoptimality in (8d). Therefore, the optimal objective functionvalue in (8a) with z∗ remains the same.

Consider again the three-node example of Fig. 1. With theAD objective using (7), we find the optimal solution to bex∗11 = 0.5, x∗21 = 0.8, x∗22 = 0.2, x∗31 = 0.4. In this case, atotal of four flow variables are nonzero at optimality. AlthoughProposition 7 calls for a second phase to find an optimalsolution x with at most D+L positive flow variables, solving(8) once was sufficient in our experience (for all the casesstudied later in Section III) as the computed optimal solutionsx∗ always consisted of less than D + L positive flows.

An important consequence of Proposition 7 is that this alsoholds for the original non-linear program (5) since we can getarbitrarily close to (5) through a series of applications of (8).

Proposition 8: If (5) is feasible, then there exists an opti-mal vertex solution with at most D+L positive flow variables.

Importantly, the D+L property is not limited to either (8)or (5); this holds for an arbitrary objective function.

Lemma 9: (generalized D + L property) Consider theoptimization problem of minimizing an arbitrary objectivefunction of the form F (y) subject to constraints (5b)-(5d) andx ≥ 0. If this problem is feasible, then there exists an optimalsolution x that has at most D + L positive flows.

Proof: Suppose that y∗ = (y∗` : ` ∈ L) is the optimal linkloads. With y∗ fixed, consider again the polytope defined inx(≥ 0) given by (9). Clearly, the polytope given by (9) hasD+L equations; thus, any of its vertices x has at most D+Lnonzero flow solutions—this follows from the same argumentas in Proposition 7. Again, by (9b), x is an optimal solutionof the considered problem.

We also note that Lemma 9 implies Proposition 1. Finally,Lemma 9 can be made more general and extended to a moregeneral form of the objective function (see Section V) and ofthe solution polyhedron.

E. Multipath Measure

We have shown that for the three objectives, MCR, LB, andAD, the maximum number of nonzero flows at an optimalvertex solution is equal to D + L, D + L − 1, D + L,respectively. Corollaries 2, 6, and Proposition 7 suggest thefollowing related measure.

DEFINITION 1. The multipath measure, MPM, is equal to themaximum percentage of demands that can have more than onepath with nonzero flow at optimal vertex solutions.

Thus, for MCR, LB, and AD problems, MPM is givenby L/D, (L − 1)/D, and L/D, respectively (maximumbeing one). It is important to note that this measure is trafficinvariant. In Table I, we report MPM (as percentage) for threenetwork topologies. The number of links in an N -node ringnetwork is simply N . The grid is a square network that hasL = 2N − 2

√N links. The 3N -net denotes a network with

L = 3N (average vertex degree 6). We use 3N -net as anupper bound on the number of links in a network since mostpractical ISP networks rarely have more than 3N links. Forinstance, among topologies in the Topology Zoo collection[11], the highest average vertex degree (2γ) was found to be4.5, i.e., the link-to-node number ratio (γ = L/N ) was 2.25.For the RocketFuel ISP topologies at the PoP level [10], thehighest average vertex degree (2γ) was reported to be 5.24,i.e., the link-to-node number ratio (γ = L/N ) was 2.62.

Assume again that all pairs of nodes in a network havetraffic, i.e., D = N(N − 1)/2. In Table I, we show thevalues of D and the MPM as the number of network nodesN is increased from 4 to 1,024. Note that a ring network hasthe least number of links of the three topologies considered;naturally, the MPM is the smallest with the ring networkcompared to the other networks. It is interesting to see thatfor a 4-node ring network, at most 66.67% of the demandscan have more than one path with nonzero flows for the MCRproblem. For the LB objective that has one path less (MPM= (L− 1)/D), the MPM drops down to 50% already for the4-node ring network.

From Table I, we also observe that as N increases, the MPMdrops to no more than 2% for N = 100, and further down tono more than 0.2% for N = 1, 024 for ring networks with theMCR and the AD objectives. Thus, a general conclusion fromthis high-level analysis based on the D+L property is that ifthe network is sparse, then the MPM can drop significantly asthe total number of nodes increases; in other words, there is nosignificant gain from having multipath routing. If a networkhas 3N links, then the MPM also follows the similar behaviorfor large N . Certainly, the gain with multipath routing is moreprominent when N is not too high.

Thus far, our discussion has been mainly focused on thetheoretical assessments based on Corollaries 2, 6, and Propo-sition 7. By revisiting the 3-node network example, we found

that, quite often, the maximum number of nonzero flowsis below what is prescribed by the theoretical assessmentexpressed by the MPM. Thus, to avoid confusion, the MPMthat is determined by solving a specific LP problem optimallywill be denoted by MPM∗ to distinguish from the theoreticalMPM. The theoretical assessment, along with the illustrationof the three-node example for all three objectives, raisesthe following key questions: How does the traffic volumeand pattern impact the number of nonzero flow variables atoptimality? How much less is this from the theoretical MPM?How does the topology factor in with traffic in this regard? Inthe rest of the paper, we attempt to answer these questions.Before going further, we point out that for the tables presentedlater, a single row entry is shown for the MPM∗ for the LBobjective regardless of the load levels since it suffices to showone entry due to traffic scaling discussed for the LB objectivein Section II-C.

III. STUDY: ALL NODE PAIR DEMANDS

In this section, we present our study for the case when allnode pairs have traffic, i.e., D = N(N − 1)/2, for a numberof topological structures and load distributions.

A. Uniform Traffic and Capacity

Here, we assume that every demand has uniform traffic, i.e.,hd = h, d ∈ D, and every pair of nodes (D = N(N − 1)/2)has this traffic volume. We also assume that the capacity isuniform, c` = c, ` ∈ L. We consider two different topologicalstructures: ring and (square) grid. In a ring topology, everydemand has two paths to use, one clockwise and the othercounter-clockwise. Ring topologies are commonly deployedin metropolitan areas. A grid topology has multiple paths be-tween each pair of nodes – the number of possible paths growswith the increase in the size of the grid. Commonly deployedoptical networks in the continental US have a rectangulargrid structure (of rings). Results for representative realistictopologies with nonuniform traffic will be presented later.

1) Ring Topology: For a ring network, we analyticallyderive the exact value of the MPM (marked as MPM∗) forthe uniform traffic case for the LB and AD objectives.

Proposition 10: Consider the LB problem (4) for a ringnetwork with odd N , uniform traffic (hd = h, d ∈ D), anduniform capacity (c` = c, ` ∈ L). Then at the (unique) optimalsolution every demand uses the single hop-based shortest pathto route its traffic. Hence,

MPM∗ = 0, r∗ = N(N−1)h8c , (10)

Furthermore, y∗` = N(N−1)h8 , ` ∈ L.

The proof is omitted due to the lack of space. There are thefollowing important observations related to this result:

1) Proposition 10 holds for the full range of traffic-to-capacity ratios h

c since the LB problem is valid for allr ≥ 0, as discussed after formulation (4).

2) Proposition 10 is also valid for AD in the approximateformulation (8) (for any h

c ), except for the formula forr∗. Thus, at optimality, for any h

c the AD objective(8a) forces single-path routing and the network is fully

N = 4 N = 9 N = 16 N = 25 N = 36 N = 49 N = 64 N = 100 N = 1024

D = N(N − 1)/2 6 36 120 300 630 1,176 2,016 4,950 523,776Ring (MCR, AD) 66.67 25.00 13.33 8.33 5.71 4.17 3.17 2.02 0.20Ring (LB) 50.00 22.22 12.50 8.00 5.56 4.08 3.13 2.00 0.20Grid (MCR, AD) 66.67 33.33 20.00 13.33 9.52 7.14 5.56 3.64 0.38Grid (LB) 50.00 30.56 19.17 13.00 9.37 7.06 5.51 3.62 0.383N -net (MCR, AD) 100.00 75.00 40.00 25.00 17.14 12.50 9.52 6.06 0.593N -net (LB) 83.33 72.22 39.17 24.67 16.98 12.41 9.47 6.04 0.59

TABLE I: MPM (in %) for different network sizes and topologies (Ring: L = N ; Grid: L = 2N − 2√N ; 3N -Net: L = 3N )

balanced: y∗` = r∗

c , ` ∈ L. This means that when theoriginal AD problem (5) is feasible (hc ≤

8ηN(N−1) , see

below) the single-path routing and the above link loadsremain optimal. Hence, F4(y∗) = 1

H

∑`∈L

y∗`c`−y∗`

=LH

r∗

1−r∗ .Consider next when N is even. In this situation, for the LB

objective, each demand will use the shortest-hop path exceptfor the demands for which each path is equal in terms of thehop distance. For the demands with both routing paths equalin the hop distance, the traffic will be equally split among thetwo paths to balance the load of the network links; there areN/2 demands that are in this situation.

Proposition 11: If N is even and the traffic and linkcapacity are uniform in a ring network, then exactly N/2 de-mands will use two paths at optimality for the load balancingobjective, and the rest will use single paths, resulting in

MPM∗ = 1N−1 , r∗ = N2h

8c(11)

and the network is fully balanced, y∗` = N2h8 , ` ∈ L.

When N is even, MPM∗ (= 1/(N−1)) is substantially smallerthan the theoretical MPM, being about half of MPM (= 2/N ).

Again, Proposition 11 holds for the full range of hc , also

for the approximate AD formulation (8). In the case of MCR,the feasible region is h

c ≤8N2 . When the MCR problem is

feasible, the flow solution of Proposition 11 is optimal for thehop count cost structure.

In general, for MCR, the actual selection of paths dependson the cost structure of the unit cost of flow and how loadedthe network is. In Table IIa, we report values of the MPM∗

obtained by solving1 the LP formulation (1) of the MCRproblem with the hop-based path cost (“MCR-H”), and theswapped path cost (“MCR-S”, i.e., where the shorter hop pathhas the hop cost of the longer hop path and vice versa). Theresults for LB/AD objectives are also included for comparison.

We consider a number of normalized load levels from 0.4to 0.95. Generally, we will refer to 0.4 as the low loadlevel, 0.6 as the moderate load level and 0.8 or higher asthe high load level. The low load level of 0.4 can alsobe interpreted as having a highly redundant capacity. Mostpractical networks attempt to maintain average load levelaround 0.6. The higher load levels are included to obtain acomprehensive understanding.

From Table IIa, we see that at a low to moderate load level,the MPM∗ = 0 most often for the MCR-H case. When N

1All LP and ILP problems were solved to optimality using CPLEX. ForAD, the optimal solutions were obtained for the PLA using (8).

is 64 or higher, the MPM∗ = 0 at all load levels. However,for the MCR-S case at a low load, the network tries tospread out using more than one path for demands, often morethan the corresponding value of the MPM∗ for the LB/ADobjectives and occasionally reaching the upper bound of theresult presented in Corollary 2, especially when N is even (seeTable I). When N is odd, the spread is much smaller than theupper bound, and sometimes the MPM∗ = 0. As expected, theMPM∗ is the highest when the number of nodes is very small.

2) Grid Topology: There are no general results for a gridnetwork for any of the objectives. Thus, here we considera number of grid sizes with uniform traffic. The results aretabulated in Table IIIa. We found that for the LB objective,the MPM∗ was either zero or close to zero when N ≥ 16. Withthe AD objective, we found that slightly more demands hadmultiple paths at optimality than for the LB objective. Whenwe considered the different load levels for the AD objective,there was no clear indication that a higher load would resultin a greater number of demands having a multipath solutionat optimality as the network size grew.

With the MCR problem, we considered two cost valuesfor ξdp, one that was based on the hop count (“MCR-H”)and the other where the unit cost was randomly generated(“MCR-R”). For MCR-R, we generated five sets of randomcost values and the results shown were the average from theseruns. In general, the MPM∗ = 0 for the MCR-H problem. Forthe MCR-R objective, as the network capacity became tighter,some demands tended to spread traffic by using multiple paths.If we compare the MPM∗ of MCR-R to the theoretical MPMshown in Table I, we find that a considerably less number ofdemands can avail of multiple paths at optimality when N ismoderately large.

B. Nonuniform TrafficReal traffic data sets, especially for large networks, are

impossible to obtain. Thus, to understand MPM at differenttraffic instances, we considered a number of non-uniformtraffic matrices as follows:• We considered a variation of the uniform traffic, where

traffic for each demand was adjusted randomly from uni-form demand h by a small percentage, i.e., h± rand()ε.The goal was to understand how small perturbationsimpact the MPM∗ compared to uniform traffic. We set thevalue of ε to 2% and 10% of h, so that the mean variationwas 1% and 5%, respectively. We report the results forthe former as there were minimal differences betweenthem in terms of MPM. This distribution is labeled asuniform-perturbed traffic distribution.

0 20 40 60 80 1000

10

20

30

Ring Topology: LB

Network Size

MP

M*(

%)

U

E−M

LN

U−P

MPM

(a) LB

020

4060

80100 0.4 0.5 0.6 0.7 0.8 0.9 11

0

10

20

30

Load Level

Ring Topology: AD

Network Size

MP

M*

(%)

U

E−M

LN

U−P

MPM

(b) AD

020

4060

80100 0.4 0.5 0.6 0.7 0.8 0.9 11

0

10

20

30

Load Level

Ring Topology: MCR−H

Network Size

MP

M*(

%)

U

E−M

LN

U−P

MPM

(c) MCR-H

020

4060

80100 0.4 0.5 0.6 0.7 0.8 0.9 11

0

10

20

30

Load Level

RIng Topology: MCR−S

Network Size

MP

M*(

%)

U

E−M

LN

U−P

MPM

(d) MCR-S

Fig. 2: MPM∗ and MPM for Ring Topology: All Objectives, Traffic Distributions and Load Levels(U: Uniform; E-M: Elephant-Mice; LN: Lognormal; U-P: Uniform-Perturbed; MPM: Theoretical Value)

N = 9 16 25 36 49 64 100LB / AD 0.00 6.67 0.00 2.86 0.00 1.59 1.01MCR-H0.4 - 0.8 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.9 0.00 0.83 0.00 0.00 0.00 0.00 0.000.95 0.00 0.83 0.00 0.16 0.00 0.00 0.00

MCR-S0.4 0.00 11.67 0.00 2.49 0.68 2.88 1.470.6 19.44 10.83 2.67 4.44 0.00 3.17 1.410.8 8.33 11.67 3.00 3.33 0.00 2.88 1.450.9 8.33 12.50 2.67 3.49 0.77 2.28 1.31

0.95 8.33 10.00 1.00 4.76 1.02 2.33 1.45

(a) MPM∗ for Ring: Uniform Traffic

N = 9 16 25 36 49 64 100LB 11.11 8.50 6.27 4.86 3.35 2.75 1.72AD0.4 0.00 0.00 0.00 0.00 0.00 0.00 0.000.6 0.00 0.83 0.00 0.00 0.00 0.00 0.000.8 4.45 6.67 1.27 2.86 0.49 1.53 0.890.9 4.45 6.67 1.27 2.86 0.51 1.51 0.900.95 4.45 6.67 1.27 2.86 0.51 1.51 0.89

MCR-H0.4 - 0.8 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.9 0.00 0.83 0.00 0.00 0.00 0.00 0.000.95 0.00 0.83 0.00 0.16 0.00 0.00 0.00

MCR-S0.4 - 0.95 25.00 13.33 8.33 5.71 4.17 3.17 2.02

(b) MPM∗ for Ring: Uniform-Perturbed Traffic

N = 9 16 25 36 49 64 100LB 2.78 6.67 0.33 2.86 0.09 1.59 1.01AD0.4 2.78 6.67 0.33 0.95 0.00 0.00 0.000.6 2.78 6.67 0.33 0.16 0.09 0.05 0.00

0.8 - 0.95 2.78 6.67 0.33 2.86 0.09 1.59 1.01MCR-H0.4 - 0.6 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.8 2.78 0.83 0.33 0.16 0.00 0.00 0.000.9 2.78 0.83 0.33 0.16 0.09 0.05 0.00

0.95 2.78 0.83 0.33 0.16 0.09 0.05 0.02MCR-S0.4, 0.5 0.00 0.00 0.33 0.16 0.68 2.98 1.45

0.6 2.78 12.50 5.00 3.65 0.09 2.33 1.580.8 8.33 12.50 1.33 4.60 0.77 3.03 1.450.9 13.89 12.50 2.33 3.97 0.68 2.68 1.29

0.95 16.67 9.17 1.00 3.81 0.94 2.08 1.43

(c) MPM∗ for Ring: Elephant-Mice Traffic

N = 9 16 25 36 49 64 100LB 7.78 3.83 2.74 2.28 1.53 1.25 0.54AD0.4 4.45 1.83 0.93 0.16 0.00 0.00 0.010.6 3.33 0.50 0.07 0.06 0.00 0.00 0.000.8 5.00 2.50 1.20 0.73 0.24 0.18 0.060.9 7.22 3.00 1.73 0.79 0.31 0.18 0.06

0.95 7.22 3.83 1.60 0.95 0.46 0.33 0.10MCR-H0.4 - 0.8 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.9 1.67 0.83 0.07 0.06 0.00 0.00 0.000.95 3.89 1.00 0.27 0.16 0.04 0.01 0.01

MCR-S0.4 2.78 1.67 2.27 3.14 2.18 1.56 0.700.6 20.00 9.67 6.00 3.84 2.50 1.90 1.100.8 20.00 9.17 5.20 3.65 2.45 1.89 1.050.9 17.78 9.67 5.33 3.62 2.33 1.85 0.98

0.95 17.22 9.50 5.20 3.78 2.23 1.78 0.93

(d) MPM∗ for Ring: Lognormal Traffic

TABLE II: MPM∗ for Ring Topology

• A specific demand was selected to have significantlyhigher traffic than the rest of the demands. This situationallowed us to to see how a large traffic stream (“ele-phant”) affected small streams (“mice”) in the rest of thenetwork in terms of the MPM. Secondly, it also allowedus to imagine the situation if there was only one demandin the network (D = 1) to start with, but the demandfor the rest of the network started to pick up with a lowtraffic amount. We refer to this traffic distribution as theelephant-mice traffic distribution.

• The third traffic distribution was based on observationsfrom backbone networks [16] that reported that trafficfollows a lognormal distribution. Using the parametervalues for the lognormal distribution given in [16] (µ =16.6, σ = 1.04), five sets of traffic data were generated.

The reported results were based on the average of the fivesets. This traffic is referred to as the lognormal traffic.

In addition to the ring and grid topologies, we consideredthree representative topologies of different sizes from real ISPnetworks: GEANT, NTT-Global, and Level-3. The GEANTtopology, discussed extensively in [22], consists of 22 nodesand 36 links (i.e., average vertex degree is 2γ = 3.27); see[17] for its actual traffic data set. NTT’s Global IP backbonetopology (NTT-Global) is selected from the Internet TopologyZoo collection [11], except that we used the latest topology;this topology has 31 nodes and 72 links (i.e., 2γ = 4.65). Thethird topology is Level-3’s topology [19] that has 95 nodesand 128 links (i.e., 2γ = 2.69). The stub nodes in NTT-Globaland Level-3 were not considered as they could be collapsed totheir parent nodes. For NTT-Global and Level-3, no traffic data

was available to us; in these two cases, we used the lognormaldistribution to generate traffic. Based on a distribution fitnesstest, we found the GEANT’s actual traffic to be also lognormal.

1) Ring Topology: First, consider uniform-perturbed traffic(see Table IIb). It is helpful to compare this against the resultfor the uniform traffic (Table IIa) to see how a small per-turbation impacts the MPM. Consider now the LB objective.For an odd number of nodes, the MPM∗ that used to be 0for uniform traffic goes up to 6.27% for uniform-perturbedtraffic for N = 25. For an even number of nodes, the MPM∗

increases from 2.86% for uniform traffic to 4.86% for uniform-perturbed traffic for N = 36. When the number of nodes Nincreases to 100, the difference is minimal (1.72% comparedto 1.01%); for comparison, the theoretical MPM is 2.00% forN = 100. For the AD objective, the number of demandswith multiple nonzero path flows is generally smaller thanthe counterpart for the LP objective. For the MCR problemwith the hop-based path cost (“MCR-H”), there was no changeobserved from uniform to uniform-perturbed traffic. For theMCR problem with swapped cost (“MCR-S”), the MPM wasthe same as the theoretical limit.

For the elephant-mice traffic (Table IIc), we found thatbecause of the elephant traffic stream, several demands withmice traffic were pushed to resort to multipath routing for theLB objective when N was odd and small. Otherwise, therewas no major difference in impact when the number of nodeswas even.

Next, consider the lognormal traffic (Table IId). While thisload distribution generally resulted in a higher MPM∗ thanuniform traffic, it was somewhat surprising to find that theMPM∗ for lognormal traffic was lower than for uniform-perturbed traffic. In Fig. 2, we present a visual comparison ofthe MPM∗ for different objectives, load levels, and networksizes.

2) Grid Topology: When traffic distribution changed fromuniform (Table IIIa) to uniform-perturbed (Table IIIb; see alsoFig. 3), a significantly higher number of demands resorted tomultipath routing for the LB and AD objectives in relativelysmall grid networks (i.e., N ≤ 25). However, this differenceessentially disappeared as the grid topology grew. For instance,when N = 100, for uniform and uniform-perturbed traffic,the MPM∗ was 0.95% and 0.75%, respectively, for the LBobjective. For the MCR problem with hop-based path cost(“MCR-H”), there was no difference as every demand usedthe shortest path at optimality for both uniform and uniform-perturbed traffic. When the unit path cost was randomlyassigned in the MCR problem (“MCR-R”), the MPM∗ wasnoticeable in small grid networks at high load levels. Forinstance, in a 4-by-4 grid network (N = 16), there were about15.33% of the demands that tended to use multiple paths atthe 95% traffic load level.

For the elephant-mice traffic (Table IIIc), we noted that theelephant traffic stream pushed several mice traffic streams toseek multipath routing when the number of nodes in a gridtopology was small, quite similar to the ring topology. Forthe MCR problem with a hop-based path cost, we noted thatat a high load level, there was a small number of demands

that split traffic to multi paths (MPM∗ = 1.67% for N = 16).However, in a large grid network (N ≥ 64), the MPM∗ =0; this meant that every demand chose the shortest path at anoptimal solution. When the grid network had lognormal traffic(Table IIId), the MPM∗ was found to be similar to uniform-perturbed traffic.

3) Topologies: GEANT, NTT-Global, Level-3: For the 22-node GEANT topology, we used three actual traffic profilesat three different times: 9:00, 12:00, and 21:00 on July 21,2005. Since the GEANT topology has 36 links, its theoreticalMPM for the LB objective is 15.15% for all pair demands.On the other hand, from our runs, we observed the computedMPM∗ to have the value of 3.03% for the three different loads(Table IV) for the LB objective. The highest value of theMPM∗ was observed to be 4.76% for MCR-R, which was stillconsiderably smaller than the theoretical value of 15.58%.

The MPM∗ values for GEANT, NTT-Global and Level-3topologies for the lognormal traffic (with µ = 16.6, σ = 1.04)are also reported in Table IV. For the LB objective, thetheoretical MPM for the NTT-Global topology was 15.27%,but we found that there were only 1.46% demands (about7 demands) that relied on multipath routing. When the trafficload was moderate for the MCR problem, only one pairsplit the traffic into multiple paths. For the AD objective,we observed that the MPM∗ values increased as the networkcapacity became tighter, and the highest value was 6.38%,which was far less than the theoretical MPM of 15.48%.

Next, consider the 95-node Level-3 topology. The averagevertex degree of this topology was 2.69, which is between theaverage vertex degree of the ring topology and grid topologyfor N = 95. For the LB objective, the MPM∗ was foundto be 0.4%, which is less than its theoretical MPM value of2.8%. The MPM∗ values at various load levels for the ADobjective fell between that of the 100-node ring topology and100-node grid topology. For the MCR problem, the value ofthe MPM∗ at maximum was observed to be 0.5% for MCR-R,which meant that about 22 out of 4,465 demands used morethan one path at an optimal solution.

Based on many cases studied here, the observed MPM∗

value was rarely found to exceed 10% for N = 25 nodesirrespective of topologies and traffic profiles. Thus, we usedN = 25 as the marker point to indicate that networks withN > 25 showed considerably less gain with multipath routing.

C. Comparison with Single-Path Routing

From the studies above, we see that only a small fractionof demands used more than one path at optimality when thenumber of nodes in the network was high. This raises a naturalquestion as to how much would we lose out if a network wereto allow only single-path routing? It is sometimes preferable toenforce single-path routing [20]. Note that single-path routingis not the same as shortest path routing. As a consideration,single-path routing may be invoked in MPLS or SDN net-works, while the shortest path routing (with the equal-costmultipath feature) is used in IP networks.

The single-path routing problems for the three TE objectivescan be formulated as integer linear programming (ILP) prob-

0 20 40 60 80 1000

10

20

30

40

Network Size

Grid Topology: LB

MP

M*(

%)

U

E−M

LN

U−P

MPM

(a) LB

020

4060

80100 0.4 0.5 0.6 0.7 0.8 0.9 11

0

10

20

30

40

Load Level

Grid Topology: AD

Network Size

MP

M*

(%) U

E−M

LN

U−P

MPM

(b) AD

020

4060

80100 0.4 0.5 0.6 0.7 0.8 0.9 11

0

10

20

30

40

Load Level

Grid Topology: MCR−H

Network Size

MP

M*(

%)

U

E−M

LN

U−P

MPM

(c) MCR-H

020

4060

80100 0.4 0.5 0.6 0.7 0.8 0.9 11

0

10

20

30

40

Load Level

Grid Topology: MCR−R

Network Size

MP

M*(

%)

U

E−M

LN

U−P

MPM

(d) MCR-R

Fig. 3: MPM∗ and MPM for Grid Topology: All Objectives, Traffic Distributions and Load Levels(U: Uniform; E-M: Elephant-Mice; LN: Lognormal; U-P: Uniform-Perturbed; MPM: Theoretical Value)

N = 9 16 25 36 49 64 100LB 0.00 0.00 0.00 0.79 0.77 1.24 0.75AD0.4 0.00 0.00 0.00 0.00 0.00 0.00 0.480.6 0.00 0.00 0.00 2.38 0.00 0.00 0.320.8 0.00 0.00 0.00 3.49 0.00 1.93 1.050.9 0.00 12.50 0.00 1.43 0.00 2.28 1.37

0.95 0.00 6.67 0.00 4.60 0.00 2.38 1.72MCR-H

0.4 - 0.95 0.00 0.00 0.00 0.00 0.00 0.00 0.00MCR-R

0.4 0.00 0.00 0.00 0.00 0.00 0.00 0.000.6 2.78 1.83 0.00 0.06 0.07 0.12 0.050.8 23.33 7.17 4.00 2.95 2.79 1.52 0.650.9 25.00 12.67 6.00 4.66 3.83 2.64 1.33

0.95 23.89 14.33 7.20 5.49 4.10 3.05 1.65

(a) MPM∗ for Grid: Uniform Traffic

N = 9 16 25 36 49 64 100LB 16.11 5.33 4.13 2.28 2.79 1.93 0.95AD0.4 0.00 0.00 0.00 0.00 0.00 0.00 0.490.6 0.00 0.00 0.00 3.49 1.57 0.00 0.290.8 21.66 14.00 8.40 2.32 3.59 1.49 1.000.9 22.22 14.67 9.20 4.38 2.33 2.00 1.190.95 21.66 14.67 9.40 3.27 3.25 2.06 1.74

MCR-H0.4 - 0.95 0.00 0.00 0.00 0.00 0.00 0.00 0.00MCR-R

0.4 3.34 0.00 0.00 0.00 0.00 0.00 0.000.6 20.55 2.33 0.73 0.73 0.29 0.14 0.060.8 26.67 10.00 5.00 3.59 2.88 1.58 0.670.9 27.78 13.50 6.47 4.98 3.90 2.65 1.330.95 27.22 15.33 7.87 5.68 4.15 3.07 1.65

(b) MPM∗ for Grid: Uniform-Perturbed TrafficN = 9 16 25 36 49 64 100LB 2.78 0.83 0.67 0.16 0.68 1.93 1.21AD0.4 2.78 0.83 0.67 0.16 0.94 0.55 0.420.6 5.56 0.83 0.67 1.27 0.77 0.50 0.140.8 2.78 1.67 1.67 0.95 1.53 1.69 1.030.9 2.78 0.83 3.67 1.11 1.45 1.39 1.370.95 2.78 0.83 2.67 1.27 1.62 1.84 1.41

MCR-H0.4 - 0.9 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.95 5.56 1.67 0.67 0.48 0.26 0.00 0.00MCR-R

0.4 0.00 0.00 0.00 0.00 0.00 0.00 0.000.6 3.89 0.00 0.00 0.19 0.19 0.22 0.030.8 3.34 0.67 0.47 0.70 0.99 1.70 0.590.9 4.45 3.50 1.53 1.18 2.13 2.91 1.330.95 11.67 2.50 1.60 1.59 2.82 3.25 1.62

(c) MPM∗ for Grid: Elephant-Mice Traffic

N = 9 16 25 36 49 64 100LB 10.56 4.67 3.47 2.64 1.87 1.57 1.12AD0.4 14.44 0.00 0.00 0.00 0.00 0.00 0.630.6 14.44 9.67 0.00 2.48 0.00 0.00 0.500.8 14.44 10.00 7.13 3.46 2.87 2.19 1.280.9 17.22 11.00 7.53 3.43 2.82 2.77 1.480.95 15.56 10.50 6.93 3.56 3.11 2.75 1.83

MCR-H0.4 - 0.95 0.00 0.00 0.00 0.00 0.00 0.00 0.00MCR-R

0.4 3.33 0.00 0.00 0.00 0.00 0.00 0.000.6 16.11 2.83 0.93 0.60 0.29 0.13 0.060.8 21.11 9.00 3.33 2.95 2.15 1.51 0.710.9 22.78 11.67 5.73 4.38 3.25 2.43 1.270.95 23.89 12.00 6.53 5.18 3.66 2.92 1.57

(d) MPM∗ for Grid: Lognormal Traffic

TABLE III: MPM∗ for Grid Topology

lems. To do this, we replaced xdp = hdudp in our models andset udp = 0/1 so that u served as the binary path selectionvariable. A measure for comparing multipath routing withsingle-path routing is certainly the MPM. For the cases wherewe found that the MPM∗ was zero, there was no gain frommultipath routing compared to single-path routing. Thus, thissection focuses on the instances when the MPM∗ > 0. Here,we use the term normalized cost overhead to represent theratio, (f∗sp−f∗mp)/f∗mp, where f∗sp represents the optimal costof the single-path routing problem and f∗mp represents theoptimal cost of the corresponding multipath routing problem.This overhead is a measure of extra cost that single-pathrouting would pay compared to multipath routing.

Consider uniform traffic on the ring topology. We knowfrom Proposition 10 that when N is odd, single-path routing

provides the optimal solution. When N is even, we have thefollowing result.

Proposition 12: If N is even and the traffic and linkcapacity are uniform in a ring network, then the optimal r∗

for the load balancing objective with single-path routing isgiven by

r∗ =

hc

⌈N2

8

⌉, if N

2 is oddhc

(N2

8 + 1), if N

2 is even.(12)

Comparing r∗ in (12) with r∗ in (11), it is easy to see thatfor a ring network with N = 16, the normalized cost overheadof single-path routing over multipath routing is 3.125%, whichgoes down to 0.08% when N = 100. That is, for a largering network, single-path routing provides nearly the same

Net: GEANT (actual traffic) GEANT NTT Level-3Load 9:00 12:00 21:00 LN LN LNLB 0.87 3.03 3.03 1.47 1.68 0.40AD0.4 2.16 2.60 3.46 3.46 1.68 0.400.6 3.03 4.33 3.90 4.16 4.78 0.530.8 3.46 3.46 4.76 5.37 5.89 0.680.9 3.03 4.33 4.33 4.59 6.15 0.65

0.95 3.03 4.33 3.90 4.24 6.38 0.63MCR-H

0.4 0.00 0.00 0.00 0.00 0.00 0.000.6 0.00 0.00 0.00 0.26 0.00 0.000.8 0.00 2.16 1.30 0.87 0.91 0.100.9 1.30 2.16 1.30 1.30 1.12 0.17

0.95 1.73 2.60 2.16 1.73 1.64 0.20MCR-R

0.4 1.73 2.16 1.30 0.00 0.00 0.000.6 3.03 2.60 2.60 1.47 0.22 0.060.8 3.46 3.46 3.46 3.64 1.51 0.270.9 3.03 4.33 3.46 5.12 2.37 0.46

0.95 3.46 4.76 4.76 5.37 2.84 0.50

TABLE IV: MPM∗ for GEANT, NTT and Level-3(LN: Lognormal)

0.4 0.5 0.6 0.7 0.8 0.9 10

10

20

30

40

Load Level

No

rma

lize

d

Ove

rhe

ad

(%

)

7:00AM12:00PM9:00PM

Greater than 40%

(a) AD Objective

Fig. 4: Normalized Cost Overhead: GEANT Topology

0.4 0.5 0.6 0.7 0.8 0.9 10

0.25

0.5

Load Level

No

rma

lize

d

Ove

rhe

ad

(%

)

NTT TopologyLevel3 Topology

µ = 16.6, σ = 1.04

2.3%

0.01%

(a) AD Objective

0.4 0.5 0.6 0.7 0.8 0.9 10

0.005

0.01

Load Level

No

rma

lize

d

Ove

rhe

ad

(%

)

NTT TopologyLevel3 Topology

µ = 16.6, σ = 1.04

(b) MCR-H Objective

Fig. 5: Normalized Cost Overhead: NTT & Level3 (lognormal)

performance level as multipath routing for uniform traffic.For the lognormal traffic, single-path routing had high costoverhead over multipath routing when the network size wassmall and almost fully loaded. For instance, the normalizedcost overhead was 24.6% for a 9-node ring network at the95% load level, which dropped down to 2.5% for N = 16.

Now, consider the MCR problem. If the path cost was basedon the hop count, there was no remarkable cost overhead forsingle-path routing with uniform or uniform-perturbed traffic.Even with the lognormal traffic, we noted about 1% normal-ized cost overhead for the nine-node ring topology at the 95%load level. If the path cost was swapped, the normalized costoverhead was about 6% for N = 9 with lognormal traffic; thisoverhead decreased to zero when N ≥ 64.

The elephant-mice traffic was somewhat different from

the other three traffic distributions, where only one demandhad very large traffic compared to the other demands. Forsmall networks (i.e., N ≤ 25) at relatively high load levels,there were no feasible solutions for single-path routing. Whenthe network was sufficiently large and the load level wasmoderate, single-path routing presented the same performanceas multipath routing.

The three objectives with the grid topology showed similarbehaviors as the ring topology with the four traffic distribu-tions, while multipath routing was found to be advantageousfor small networks.

For the GEANT topology with the LB objective, the nor-malized cost overhead of single-path routing over multipathrouting was found to be 10.0% for the 9:00 dataset to, 13.5%for the 12:00 dataset to 2.8% for the 21:00 dataset. With theAD objective, at low load levels (i.e., 0.4), the optimal costfor single-path routing was about 4% more than the optimalcost of multipath routing while as the load level increased,the cost overhead increased. For the AD objective (Fig. 4),when the load level was less than 0.6, the normalized costoverhead was observed to be zero; when the load level wasgreater than 0.8, single-path routing problem was infeasibleat either 9:00 or 12:00, but was feasible at 21:00. GEANTbeing a smaller topology (N ≤ 25), such differences were notsurprising. Secondly, the observed differences were for highload levels.

We compared the overhead cost for NTT-Global and Level-3topology using lognormal traffic in Fig. 5. The cost overheadwas found to be only a few percent more or less for NTT-Global and Level3, both of which are larger networks.

D. Fully Mesh Topology

We now briefly comment on fully mesh networks, in whichthe number of links are the same as the number of demands.This means that, for MCR and AD, the value for the MPM is100%, for LB it is slightly below 100% (L − 1 instead of Lused in computing MPM). This means that every demand canpotentially require multiple paths at optimality.

Consider the case of uniform traffic and capacity. It is easyto see that every demand uses its direct link path, i.e., exactlyone path, for traffic flow for the LB objective at optimality,and the network is fully balanced. Consequently, the MPM∗

= 0 with r∗ = hc . Therefore, the solution of the LB objective

also holds for the AD objective (8a) with the MPM∗ = 0. Ifthe unit path cost is based on the hop count, then the MCRproblem also results in the MPM∗ = 0. In other words, single-path routing is the optimal solution for a fully mesh networkwith uniform traffic and capacity.

As discussed earlier, currently deployed network topologiesfor IP and optical networks are nowhere close to a fullymesh topology. There are indeed real examples of fully meshbackbone communication networks. For instance, dynamic callrouting [8] in several backbone networks was deployed overfully mesh topologies during the 1980s and the 1990s. Inthis environment, calls were allowed to take at most two-linkpaths for alternate routing. Because of the blocking nature in acircuit-switched routing network, resulting in a concave mini-

mization objective, the TE problem was different in nature [8].However, as shown in [7], the performance (call throughput)of dynamic call routing is bounded by the maximum flowmulti-commodity network flow problem, which is similar tothe MCR problem and has D + L constraints, thus satisfyingthe D + L property.

Using an actual data set from Sprint’s 43-node dynamic callrouting network [13], we solved this maximum flow problemfor different traffic loads during the day by varying how tightthe network capacity was. We observed that on average 19%of the demands use multipath routing at optimality, while ina few instances as high as 47% of the demands resorted tomultipath routing. While these values are below the theoreticalMPM of 100%, they give us a glimpse that in a fully meshtopology, a significant number of demands at a particular timeinstant are able to take advantage of multiple paths to spreadtraffic to improve network performance.

IV. STUDY ON FAT-TREE DATA CENTER TOPOLOGY

Data center networks have structured topologies and moreimportantly, traffic is between a certain pair of specificswitches, not between every pair of switches. A popular intradata center network is represented by a fat-tree topology asdiscussed in [2] consisting of k pods (with k up to 48) withthree layers of switches: edge switches, aggregation switchesand core switches. Since we are interested in the TE problem,we consider the traffic to be between any pair of edge switches(ignoring hosts connected to them). Fat-tree topologies aretypically referred to in terms of the number of pods. In a k podfat-tree topology, there are k switches in each pod arrangedin two layers of k/2 switches, one layer for edge switchesand the other for aggregation switches. Each edge switch isconnected to k/2 aggregation switches. There are (k/2)2 coreswitches each connected to k pods. It is easy to see that thereare a total of N = k2+(k/2)2 = 5

4k2 switches. There are k/2

edge switches in each pod; thus, for k pods, there are a totalof k2/2 edge switches. Therefore, the number of demands isD = (k2/2)(k2/2−1)/2 = k4

8 −k2

4 . Each aggregation switchis connected to k/2 edge switches and k/2 core switches;thus, the total number of links, L = 2k(k/2)2 = k3/2. Fork = 4, the fat-tree topology is shown in Fig. 6. Here, the 8edge switches (nodes) at the bottom, numbered 1 to 8, havetraffic amongst them; thus, here D = 28(= 8 × 7/2). This4-pod fat-tree topology has L = 32 links.

In Table V, we report results for the fat-tree topology forthree values of the number of pods for the LB objective usinguniform and lognormal traffic (averaged over five sets). All

9 10 11 12 13 14 15 16

1 2 3 4 5 6 7 8

17 18 19 20

Edge

Aggregation

Core

Fig. 6: Fat-Tree Topology with k = 4 Pods

k N D L MPM MPM∗ COH MPM∗ COH(U) (U) (LN) (LN)

4 20 28 32 100.00 35.71 14.29 15.00 0.536 45 153 108 69.93 39.87 5.88 10.85 0.368 80 496 256 51.41 28.02 3.23 9.47 0.31

TABLE V: Fat-Tree Topology: MPM and MPM∗ for LB(U:=uniform, LN:=lognormal; COH:=Cost Overhead in %)

links were set to 10GE capacity. We can clearly see thatthe actual MPM∗ is significantly lower than the theoreticalMPM. Although, we found MPM∗ for lognormal traffic tobe around 9% for k = 8, the cost overhead of single-pathrouting compared to multipath routing was small. It is alsoeasy to see that since L = O(k3) and D = O(k4), MPM→ 0as k increases. That is, if a very large fat-tree data centertopology is used (such as k = 48 reported in [2]), the gainfrom multipath routing is expected to be minimal.

V. RELATED WORK

Benefits of multipath routing for wide-area networks havebeen shown in a number of works; for example, see [4],[9]. However, topologies studied in these works fall into ourobservation that multipath routing remains beneficial whenN ≤ 25. Similarly, [27] showed benefits for fully meshnetworks, which aligns with our observation discussed inSection III-D. A comparison of the LB and AD objectiveswas performed in [23].

In [10], TeXCP, a distributed multipath TE approach, waspresented. This heuristic was observed to use, on average, fourpaths or more for each demand. This work also reported thatthe TeXCP cost was a few percent higher than the optimalvalue when compared for the LB objective. For the PoP-level RocketFuel topologies studied in [10], we calculated thetheoretical MPM for the LB objective and found the MPMto vary from 15.6% for the 22-node, 37-link Ebone topologyto 4.5% for the 115-node, 296-link AT&T’s topology. Thismeans that it is not necessary to have multiple paths for everynode pair at optimality for these topologies while the gainis mostly prominent when N ≤ 25. An important result fromlinear programming is that the optimal solution is attained at avertex of the polytope described by the constraints of the linearprogram associated with the TE problem. If a solution is off bya few percent from the optimal, it is then either in the inside ofthe polytope or on one of the faces of the polytope where manynonzero (non-optimal) solutions are possible. To illustrate thispoint, consider again the 3-node example discussed earlier inFig. 1 for the LB objective that resulted in one demand to usesingle-path routing at optimality. If the traffic for this demand(demand 3) is forced to be split along the two paths as 0.4−εon the direct path and ε on the alternate path, then r willincrease from 0.75 (the optimal value) to 0.75 + ε. That is, byincreasing the cost of the network away from the optimal, itis possible to find a near optimal solution that uses multipathrouting for all demands. In other words, the TeXCP heuristicis likely to have picked such a near optimal solution resultingin choosing many paths. The case of the benefit of multipathrouting presented in [14] through a heuristic also falls into the

same category.Two variants of the LB optimization problem formulated in

Section II-C were studied in [3]. In the first variant, allowablerouting paths cannot have the length (computed according tosome given link weights) exceeding a given bound. The secondvariant limited the total number of positive flows. It was shownthat both cases were NP-hard. While this is true, when thelength of the paths is limited only by the number of hops,then the first problem is polynomial since path generationis polynomial. Also, in the light of Corollary 6, the secondvariant is of limited importance.

The paper by Wang et al. [24] addressed the cross-layerproblem of routing paths and TCP congestion-controlled trans-mission rates by maximizing the utility objective for usersattempting multipath routing; our work complements theirs byconsidering a traffic engineering perspective. Lemma 9 can begeneralized to arbitrary functions of the form F (y,X), whereX = (Xd : d ∈ D) with Xd =

∑p∈Pd

xdp. That is, Lemma 1in [24] that assumed decreasing and strictly convex functionsof the form F (y,X) is a special case of our Lemma 9.

VI. FINAL DISCUSSIONS AND REMARKS

In this work, we first highlighted the little-known D + Lproperty for the minimum cost routing (MCR) problem. Wethen showed the extension of this property to the load bal-ancing (LB) and minimum average delay (AD) objectives.In particular, Lemma 9 is a novel result. The results forthe LB objective also hold when the objective is changedto maximizing the minimum residual bandwidth over all thelinks. When the network potentially has demands between allnode pairs, this set of properties shows the diminishing returnof multipath routing as the number of nodes increases. Ourstudy considering a number of networks and traffic patternsshow that the actual value MPM∗, in reality, is much smallerthan the theoretical MPM. In general, we found that among theobjectives, the AD objective distributed more flows to multiplepaths than the other objectives at high load levels. For fat-tree data centers, the advantage from multipath routing alsodecreases when the number of pods increases.

Our findings counter the intuitive belief that multipath rout-ing is capable of effectively shaping link load distribution. Itis also commonly believed that multipath routing is beneficialfor failures. When a link fails, the topology has one less link.At this instant, the paths selected may change, but not theD + L property.

Benefits in traffic efficiency expected from multipath routingobserved by others are often for heuristic methods, not for anoptimal solution. Such observations are explainable based onresults presented for near optimal solutions that reside insidethe polytope of the constraints of the linear program associatedwith a TE problem, rather then from being at the vertexsolution at optimality. Alternately, where the benefits havebeen shown at optimality, they are for N ≤ 25. Our resultsare counter-intuitive as one expects the benefit to grow as thenetwork size grows. On the other hand, the family of D + Lproperties shows that the benefit decreases as the network sizegrows with L� D. We emphasize that solving a TE problem

to optimality is important to avoid being potentially misled bya near optimal solution.

REFERENCES

[1] R. K. Ahuja, T. L. Magnanti, and J. B. Orlin, Network Flows: Theory,Algorithms, and Applications. Prentice-Hall, 1993.

[2] M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commodity datacenter network architecture,” in Proc. of ACM SIGCOMM 2008.

[3] R. Banner and A. Orda, “Multipath routing algorithms for congestionminimization,” IEEE/ACM Trans. Ntwking, vol. 15, pp. 413–424, 2007.

[4] X. Chen, M. Chamania, A. Jukan, A. C. Drummond, and N. L. S.da Fonseca, “On the benefits of multipath routing for distributed data-intensive applications with high bandwidth requirements and multido-main reach,” in 7th Annual Communication Networks and ServicesResearch Conference, 2009. CNSR’09. IEEE, 2009, pp. 110–117.

[5] B. Fortz and M. Thorup, “Internet traffic engineering by optimizingOSPF weights,” in Proc. IEEE INFOCOM’2000, 2000, pp. 519–528.

[6] L. Fratta, M. Gerla, and L. Kleinrock, “The flow deviation method:An approach to store-and-forward communication network design,”Networks, vol. 3, pp. 97–133, 1973.

[7] R. Gibbens, F. Kelly, and P. Key, “Dynamic alternative routing –modelling and behaviour,” in Proceedings of the 12th InternationalTeletraffic Congress, 1988, pp. 1019–1025.

[8] A. Girard, Routing and Dimensioning in Circuit-Switched Networks.Addison-Wesley, 1990.

[9] S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh,S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Urs H olzle,S. Stuart, and A. Vahdat, “B4: Experience with a globally-deployedsoftware defined WAN,” in Proceedings of the ACM SIGCOMM 2013conference on SIGCOMM. ACM, 2013, pp. 3–14.

[10] S. Kandula, D. Katabi, B. Davie, and A. Charny, “Walking thetightrope: Responsive yet stable traffic engineering,” in ACM SIGCOMMComp. Comm. Rev., vol. 35, no. 4. ACM, 2005, pp. 253–264.

[11] S. Knight, H. X. Nguyen, N. Falkner, R. Bowden, and M. Roughan, “TheInternet topology zoo,” IEEE JSAC, vol. 29, pp. 1765–1775, 2011, seealso http://topology-zoo.org/.

[12] L. Lasdon, Optimization Theory for Large Systems. Macmillan, 1970.[13] D. Medhi, “A unified approach to network survivability for teletraffic

networks: Models, algorithms and analysis,” IEEE Trans. on Communi-cations, vol. 42, pp. 534–548, 1994.

[14] P. Merindol, J.-J. Pansiot, and S. Cateloin, “Improving load balancingwith multipath routing,” in Proceedings of ICCCN’2008. IEEE.

[15] A. Nucci, B. Schroeder, S. Bhattacharyya, N. Taft, and C. Diot, “IGP linkweight assignment for transient link failures,” in Proc. 18th InternationalTeletraffic Congress (ITC18), Sep. 2003, pp. 321–330.

[16] A. Nucci, A. Sridharan, and N. Taft, “The problem of synthetically gen-erating IP traffic matrices: initial recommendations,” ACM SIGCOMMComp. Comm. Rev, vol. 35, no. 3, pp. 19–32, 2005.

[17] S. Orlowski, R. Wessaly, M. Pioro, and A. Tomaszewski, “SNDlib 1.0—survivable network design library,” Networks, vol. 55, no. 3, pp. 276–286, 2010, (datasets available at http://sndlib.zib.de/).

[18] M. Pioro and D. Medhi, Routing, Flow, and Capacity Design inCommunication and Computer Networks. Morgan Kaufman, 2004.

[19] J. P. Rohrer, E. K. Cetinkaya, M. J. Alenazi, and J. P. G. Sterbenz, “Re-silinets topology-map viewer,” http://www.ittc.ku.edu/resilinets/maps/.

[20] G. Swallow, S. Bryant, and L. Andersson, “Avoiding equal cost multipathtreatment in MPLS networks,” Internet RFC 4928, June 2007.

[21] J. Tomlin, “Minimum-cost multicommodity network flows,” OperationsResearch, vol. 14, no. 1, pp. 45–51, 1966.

[22] S. Uhlig, B. Quoitin, J. Lepropre, and S. Balon, “Providing public in-tradomain traffic matrices to the research community,” ACM SIGCOMMComp. Comm. Rev., vol. 36, no. 1, pp. 83–86, 2006.

[23] K. S. Vastola, “A numerical study of two measures of delay for networkrouting,” Master’s thesis, Univ. of Illinois at Urbana-Champaign, 1979.

[24] M. Wang, C. W. Tan, W. Xu, and A. Tang, “Cost of not splitting inrouting: Characterization and estimation,” IEEE/ACM Transactions onNetworking, vol. 19, no. 6, pp. 1849–1859, 2011.

[25] J. M. Wozencraft, R. G. Gallager, and A. Segall, “Data networkreliability – first annual report,” Electronic Systems Lab, MIT, Tech.Rep. ESL-IR-677, July 1976.

[26] B. Yaged, “Minimum cost routing for static network models,” Networks,vol. 1, pp. 139–172, 1971.

[27] R. Zhang-Shen and N. McKeown, “Designing a predictable Internetbackbone with valiant load-balancing,” in Quality of Service–IWQoS2005. Springer, 2005, pp. 178–192.