Journal of Network and Computer Applications 34 (2011) 252–261

Contents lists available at ScienceDirect

Journal of Network and Computer Applications

1084-80

doi:10.1

� Corr

E-m

shenh@

journal homepage: www.elsevier.com/locate/jnca

Randomized load balancing strategies with churn resilience inpeer-to-peer networks

Song Fu a,�, Cheng-Zhong Xu b, Haiying Shen c

a Department of Computer Science and Engineering, University of North Texas, Denton, TX 76207, USAb Department of Electrical and Computer Engineering, Wayne State University, Detroit, MI, USAc Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA

a r t i c l e i n f o

Article history:

Received 27 August 2009

Received in revised form

27 April 2010

Accepted 14 July 2010

Keywords:

Peer-to-peer systems

Randomized probing

Load balancing

Heterogeneous and bounded node capacity

Churn

45/$ - see front matter & 2010 Elsevier Ltd. A

016/j.jnca.2010.07.011

esponding author.

ail addresses: song.fu@unt.edu (S. Fu), czxu@w

clemson.edu (H. Shen).

a b s t r a c t

The objective of load balancing in peer-to-peer (P2P) networks is to balance the workload of peer nodes in

proportion to their capacity so as to eliminate performance bottlenecks. It is challenging because of the

dynamic nature in overlay networks, the time-varying load characteristics, and the inherent load imbalance

caused by consistent hashing functions. It is known that simple randomized load balancing schemes can

balance load effectively while incurring only a small overhead in general parallel and distributed computing

contexts. Existing theoretical works which analyze properties of randomized load balancing schemes cannot

be applied in the highly dynamic and heterogeneous P2P systems. In this paper, we characterize the

behaviors of randomized load balancing schemes in a general P2P environment. We extend the supermarket

model by investigating the impact of node heterogeneity and churn on load distribution in P2P networks.

We prove that by using d-way random choices schemes, the length of the longest queue in a P2P system

with heterogeneous nodes and churn for dZ2 is c � log logn=logdþOð1Þ with high probability, where c is a

constant. Our results have wide applicability and are of interest beyond the specific applications.

& 2010 Elsevier Ltd. All rights reserved.

1. Introduction

Peer-to-peer (P2P) networks have become, in a short period oftime, one of the fastest growing and most popular Internetapplications. An important class of P2P overlay networks is thedistributed hash tables (DHTs) that map keys to peer nodes basedon a consistent hashing function. Representatives of DHTs includeChord (Stoica et al., 2003), Pastry (Rowstron and Druschel, 2001),Tapestry (Zhao et al., 2004), CAN (Ratnasamy et al., 2001), andCycloid (Shen et al., 2006). In a DHT, each node and key has aunique ID, and a key is mapped to a node according to the DHTdefinition. The ID space of a DHT is partitioned among nodes andeach of them is responsible for the keys whose IDs are locatedwithin its ID space. An important goal in designing DHTs is toachieve a balanced partition of the hash space among peer nodes.It is often desirable that each node takes responsibility for aportion of the hash space which is proportional to its processingcapacity, measured in terms of its processor speed, availablebandwidth, and/or storage capacity. This property should besustained as nodes join and leave the system. A similar goal isdesirable in unstructured P2P networks as well.

However, consistent hashing (Stoica et al., 2003) produces abound of O(log n) imbalance of key distribution among peer

ll rights reserved.

ayne.edu (C.-Z. Xu),

nodes, where n is the number of nodes in a P2P network. Thingsare even worse in unstructured P2P systems, where no commonlyaccepted load distribution mechanism is supported. In addition,users may query geographically adjacent nodes and those thathave popular files. These lead to even more imbalanced distribu-tion of workload in the network. When a node becomesoverloaded, it cannot store any additional files or respond to userqueries any more, which affects the system utilization and usersatisfaction. To balance load among peer nodes in a network,lightly loaded peers should be selected to store files or servequeries. Load balancing in P2P networks is an important topic andmany related works have been conducted recently (Tewari andKleinrock, 2006; Shen and Xu, 2006; Zheng et al., 2006; Godfreyand Stoica, 2005; Bienkowski et al., 2005).

It is known that simple randomized load balancing schemescan balance load effectively while incurring only a small overheadin general parallel and distributed computing environments (Xuand Lau, 1997), which makes such schemes appealing for practicalsystems. The approach of multiple random choices in P2Pnetworks was used in Shen and Xu (2006), Godfrey and Stoica(2005), Kenthapadi and Manku (2005), Zhu and Hu (2005), andByers et al. (2003). Several peer nodes are probed and the onewith the least load is selected to store a file on it or service a userquery. Random choice algorithms are scalable and they require asmall number of control messages and data structures (Shen andXu, 2006; Byers et al., 2003). More importantly, they work well inP2P systems with churn, a situation where a large percentage of

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261 253

nodes join and leave continuously and rapidly, which leads to anunpredictable network size.

To theoretically analyze the effectiveness of random-choiceschemes for balancing workload in distributed systems, research-ers have proposed several approaches. Azar et al. (2000)introduced a layered induction method, by which the randomchoice problem was modeled by balls-and-bins. It provides nearlytight results. The witness tree approach was proposed by Cole et al.(1998) to tackle the random-choice problem. The probability of acertain event can then be bounded by the probability ofoccurrence of a witness tree. Generally speaking, the witness treeapproach involves the most complexity, and it has been proved tobe the most challenging one in obtaining tight results. The fluid

limit model (Mitzenmacher, 2001; Mitzenmacher et al., 2002)characterizes system dynamics by constructing a family ofdifferential equations, which makes this approach simple yetflexible. In cases where the system dynamics can be modeled bythis method, the differential equations generate very accuratenumerical results.

However, these theoretical works only analyzed systems withcompute nodes having homogeneous and infinite capacities.Moreover, node churn, a defining characteristic of P2P networks,is not modeled by the existing approaches. As a result, we cannotconclude directly that the performance bounds derived in thoseworks will still be valid in P2P networks. In this paper, we analyzethe dynamic behavior of randomized load balancing algorithms ina general P2P network, where peer nodes join and leave thenetwork at runtime and they have heterogeneous and boundedcapacities.

We model a dynamic P2P system as follows. Load queriesarrive as a Poisson stream to a collection of n peer nodes, where n

is a random variable reflecting node churn. Nodes are hetero-geneous with bounded capacity. For each query, a number of d

nodes are chosen independently and uniformly at random, andthe query is queued at the node which currently accommodatesthe fewest number of queries. We refer to such multiple choiceprocessing as d-way probing. Queries arrive to peer nodes at ratel which is relative to the node population. They are servedaccording to the FIFO protocol, and the service time for a query isexponentially distributed with mean 1.

We extend the supermarket model (Mitzenmacher, 2001) toformulate behaviors of the preceding dynamic system in generalcases and to derive system properties. We are interested incharacterizing the average response delay and the maximum loadon active nodes. However, quantifying these metrics in a generalP2P system is nontrivial. It is difficult to find closed-formsolutions to the differential equations describing the systemdynamics, after we remove certain constraints, such as staticsystem configuration, and homogeneous and infinite nodecapacities. Instead of solving the equations directly, we studythe lower and upper bounds of state variables at equilibriumpoints with reference to those in special cases. Based on thesebounds, we quantify the average response delay and themaximum load, and derive the following properties of d-wayrandomized probing in P2P networks.

Theorem 1. For any fixed time interval I, the expected time that a

query spends in a dynamic P 2P system with d-way randomized

probing (dZ2), denoted by TdðlÞ, over interval [0, I] satisfies that

TdðlÞ=logT1 is close to að1=logdÞ, for l close to 1, where a is a

constant whose value depends on capacities of peer nodes and the

change rate of the node population.

Theorem 2. For any fixed time interval I, the length of the longest

queue in the dynamic P 2P models with d-way randomized probing

(dZ2) over interval [0, I] is c log log nþOð1Þ with high probability

(1�Oð1=nÞ), where c is a constant depending on capacities of peer

nodes, d and the arrival rate of queries, and the form of the O(1) term

depends on I and some constants.

The theorems show that two-way randomized probing isasymptotically superior to one-way probing. However, byincreasing the number of choices further, efficiency of the loadbalancing algorithms does not improve significantly. These resultsare consistent with the findings by the supermarket model(Mitzenmacher, 2001) in special case where the number ofservers is fixed, and their capacities are homogeneous andinfinite. We also conduct experiments on a P2P network.Experimental results confirm the correctness of our findings.Although the randomized probing algorithms for load balancingare designed and analyzed within the context of P2P networks,the results have wide applicability and are of interest beyond thespecific applications.

The remainder of this paper is organized as follows. The basicsupermarket model is briefly described in Section 2. Section 3formulates d-way randomized probing in P2P networks. Weinvestigate the influences of node capacity in Section 3.1 and nodechurn in Section 3.2. By analyzing the equilibrium points of a P2Psystem, we quantify the expected time of a query in the systemand the length of the longest queue among peer nodes.Experimental results are shown in Section 4. Section 5 presentsthe related work. Conclusions and remarks on future works arepresented in Section 6.

2. Basic supermarket model

A load balancing scheme distributes user requests or storageloads among compute nodes and avoids hot spots. Mitzenmacher(2001) presents a supermarket model based on differentialequations to analyze both static and dynamic load balancingstrategies. In this section, we briefly describe this model, and inthe subsequent sections we will present our extension of themodel to formalize randomized probing algorithms for loadbalancing in general P2P systems.

Supermarket model analyzes properties of randomized loadbalancing strategies in a special distributed environment. In thisenvironment, user requests arrive as a Poisson stream at acollection of servers. For each request, some constant number ofservers are chosen independently and uniformly at random withreplacement from the servers, and request waits for service at theserver which currently accommodates the fewest requests.Requests are served according to the FIFO protocol, and the servicetime for a request is exponentially distributed with mean 1.Three underlying assumptions (Mitzenmacher, 2001) are: (a)unbounded server capacities; (b) static server configuration; and(c) homogeneous servers. The author derived the average time of arequest staying in the system and the maximum workload of theservers by solving the differential equations of system states.

3. Randomized probing in general P2P systems

In large-scale P2P systems, a great number of peer nodes shareresources and send queries to each other. More often than not,they have heterogeneous configurations of storage capacity andprocessing speed. In addition, dynamics is a defining character-istic of P2P networks, with nodes joining and leaving the networkfrequently. Load balancing in such large-scale and dynamicdistributed environments is challenging. Obtaining the capacityinformation of all active nodes before dispatching jobs to the most

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261254

lightly loaded nodes is expensive. Randomized probing is aremedy to this problem.

By applying randomized probing algorithms, we make dis-patch decisions based on the load status of a small number ofnodes that are selected randomly. In this way, the number of loadquery messages that are exchanged is reduced significantly. Thescalability of such algorithms is assured because the number ofcontrol messages for each decision making is almost constanteven when the system scale expands. However, theoreticallyanalyzing behaviors of randomized load balancing algorithms ingeneral cases is challenging. Node heterogeneity and churnmake the problem intractable. In this section, we extendthe supermarket model to formulate behaviors of randomizedprobing algorithms in general P2P networks and to quantifysystem dynamics with regard to the node workload and averageresponse time based on our model. Our extension is made intwo orthogonal dimensions: server capacity (from the homo-geneous and unbounded case to the heterogeneous and boundedcase) and node dynamics (from static configuration to dynamicconfiguration).

3.1. Heterogeneous and bounded node capacity

In the basic supermarket model, as described in Section 2,servers are modeled to be homogeneous with unboundedcapacity. However, in real-world P2P systems, peer nodes havelimited and different storage capacity and processing speed. Inthis section, we analyze behaviors of random-choice algorithmsfor load balancing in P2P networks where nodes have hetero-geneous and bounded capacity. We extend the basic supermarketmodel in two steps: bounding node capacity in a homogeneousenvironment (Section 3.1.1) and then allowing heterogeneousnode capacity (Section 3.1.2). Here we assume a P2P network hasstatic composition, where the number of peer nodes in thenetwork is N, a constant. Node churn or dynamic composition willbe studied in Section 3.2.

3.1.1. Homogeneous and bounded case

In a homogeneous P2P system with bounded node capacity, weuse C to denote the uniform capacity of peer nodes. The value of C

is set to the maximum number of queries that a node can queue atruntime. When a node receives a query from a peer, it serves thequery if there is extra capacity to handle it. Otherwise, it dropsthe query by its admission controller. Therefore, we adopt thesaturation policy as follows. A query is turned down when all ofthe d nodes, selected randomly and independently, are saturated,i.e. their load equals to their capacity.

Let ni(t) denote the number of nodes queuing i queries at timet; miðtÞ ¼

PCk ¼ i nkðtÞ, i.e. the number of nodes queuing at least i

queries at time t; piðtÞ ¼ niðtÞ=n be the fraction of nodes that havequeues of size i; siðtÞ ¼

PCk ¼ i pkðtÞ ¼miðtÞ=n be the tails of the pi(t).

We drop the reference to t in the notation where the meaning isclear. The si value is more convenient to work with than the pi

value. In an empty system, which corresponds to one with noquery, s0¼1 and si¼0 for 1r irC. The expected number ofqueries per node at any time t is

PCi ¼ 1 siðtÞ, and it is finite because

each node can queue at most C queries at a time.The state of the system at any given time can be represented

by a finite vector ~s ¼ ½s0,s1, . . . ,sC �. For each value of query size, itcontains the corresponding number of nodes queuing that value.We can derive the maximum load of peer nodes and the averageresponse time of queries, based on the dynamics of this stateinformation. This resulting model can be considered as a Markovchain on the above state space.

We now extend the basic supermarket model to formulate andanalyze randomized probing among peer nodes with homoge-neous and bounded capacity. The time evolution of the P2Psystem is specified by the following set of differential equations:

_si ¼ lðsdi�1�sd

i Þ�ðsi�siþ1Þ for ioC,

_sC ¼ lðsdC�1�sd

C�sC ,

s0 ¼ 1,

8><>: ð3:1Þ

where _si denotes dsi/dt.Let us explain why we have Eq. (3.1). In a P2P network with N

nodes, we will determine the expected change of the number ofnodes with at least i queries over a small period of time dt. Theprobability that a query arrives during this period is lNdt, and theprobability that an arriving query is dispatched to a node queuingi �1 queries is si�1

d�si

d, i.e. all of the d nodes chosen by the newquery have queues of size at least i�1, but not all of size at least i.The probability a query leaves a node with queue size i in thisperiod is N (si �si + 1) dt, for ioC. Because each node can serve nomore than C queries at a time, the probability a query leaves anode with queue size C in this period is NsC dt.

Next, we try to find the equilibrium points of (3.1). At theequilibrium points, the volume of incoming queries equals thevolume of outgoing queries, i.e. _si ¼ 0. For a special case d¼1,system (3.1) becomes stable at states

pi ¼li�lCþ1

1�lCþ1for 1r irC:

We denote the expected time that a query spends in the P2Psystem with homogeneous and bounded-capacity nodes by TdðlÞ.As we mention before, the probability that an incoming queryarriving at time t becomes the ith query in the queue issi�1(t)d

�si(t)d. Therefore, the expected time a query that arrivesat time t spends in the system is TdðlÞ ¼

PCi ¼ 1 iðsi�1ðtÞ

d

�siðtÞdÞ ¼

PC�1i ¼ 0 siðtÞ

d�CsCðtÞ

d. For d¼1, it is clear that at theequilibrium point,

T1ðlÞ ¼XC�1

i ¼ 0

si�CsC ¼1�ðCþ1ÞlC

þClCþ1

ð1�lÞð1�lCþ1Þ

:

Then, we consider the convergence of sequence fsiðtÞgCi ¼ 0 for dZ1.

Definition 1. A sequence fxig is said to decrease doublyexponentially if and only if there exist positive constants N,ao1, b41, and g such that for iZN, xirgab

i

.

Then, we show that every trajectory of the system (3.1)converges to a fixed point.

Corollary 2. Suppose there exists some j such that sj(0)¼0. Then the

sequence fsiðtÞgCi ¼ 0 decreases doubly exponentially for all tZ0,

where the associated constants are independent of t.

Proof. According to the definition of sequence fsiðtÞgCi ¼ 0, it is clear

that s0Zs1Z � � �ZsC , i.e. a monotone decreasing sequence. Wefirst increase si(0) such that sið0Þ ¼ si�1ð0Þ�e, where e is a small

constant. Let v¼maxðsið0Þ � l�ðdi�1Þ=ðd�1Þ

Þ1=di

. In the original system

siðtÞrsið0Þ for all tZ0. Then, siðtÞrlðdi�1Þ=ðd�1Þ

� vdi. Based on the

result in Mitzenmacher (1997), we conclude that fsiðtÞgCi ¼ 0

decreases doubly exponentially for all tZ0. Next, we further findthe upper and lower bounds of the equilibrium points. &

Sequence fsiðtÞgCi ¼ 0 decreases doubly exponentially to a fixed

point. Let ~p ¼ ½p0,p1, . . . ,pC � denote the equilibrium points ofstates fsig in the system (3.1). We now examine the expected timethat a query spends in the homogeneous and bounded-capacityP2P system.

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261 255

Theorem 3. For lA ½0,1� and dZ2, TdðlÞraðlogT1ðlÞÞ, where a is a

constant whose value depends only on d and C. Moreover,

liml-1�

TdðlÞlogT1ðlÞ

¼C

logC

2logd

: ð3:2Þ

Proof. First, we prove pirlðdi�1Þ=ðd�1Þ by induction. For i¼ 0,

p0 ¼ lðdi�1Þ=ðd�1Þ

ji ¼ 0 ¼ 1. For 0r irk, we assume pirlðdi�1Þ=ðd�1Þ.

Then for i¼k+1, we compare the equilibrium points of (3.1) andthose of the following unbounded-capacity system

_si ¼ lðsdi�1�sd

i Þ�ðsi�siþ1Þ, iZ1,

s0 ¼ 1:

(ð3:3Þ

Let ðp̂ iÞ denote the equilibrium points of (3.3). In Mitzenmacher

(1997), it was proved that p̂i ¼ lðdi�1Þ=ðd�1Þ. Then, we only need to

prove that pir p̂ i for 0r irC, which are as follows. From (3.3),

we have p̂ i ¼ lp̂di�1, iZ1. According to (3.1), pi ¼ lðpd

i�1�pdCÞ,

1r irC. Because pi, p̂ iZ0, pirlpdi�1. Based on the assumption

pir p̂ i for iA ½0,k�, we get pkþ1rlpdk rlp̂d

k ¼ p̂kþ1. Therefore, the

equilibrium points pirlðdi�1Þ=ðd�1Þ for 1r irC. On the other end,

we prove piZap̂bi , where a¼ l1=ðd�1Þ and b¼ 2�logð1þlÞ=logl.

The induction step is as follows: assume piZap̂bi for 0r irk�1,

and then according to (3.1) pkþlpdC ¼ lpd

k�1 ) lpdk�1 rpkþ

lpdk r ð1þlÞpk. Thus, pkZðl=ð1þlÞÞpd

k�1Zl= ð1þlÞ � ad � p̂bdk�1.

Because p̂k ¼ lp̂dk�1 from (3.3), we have pkZ1=ð1þlÞ�

ad�1=lb�1� ap̂b

k ¼ ð1=ð1þlÞlb�2Þap̂b

k ¼ ap̂bk .

Based on the result that ap̂bi rpir p̂ i for 0r irC, we can

calculate the upper and lower bounds of TdðlÞ.TdðlÞ ¼

PCi ¼ 1 iðpi�1ðtÞ

d�piðtÞ

dÞ ¼

PC�1i ¼ 0 pd

i �CpdC ¼

PCi ¼ 1 pi. There-

fore, aPC

i ¼ 1 p̂bi rTdðlÞr

PCi ¼ 1 p̂ i. When l-1�, pi tends to be

lgðdi�1Þ=ðd�1Þ, where g is a constant within ½1, 3

2�. Therefore,

TdðlÞ ¼PC

i ¼ 1 lgðdi�1Þ=ðd�1Þ. Because

QC�1i ¼ 0ð1þl

di

þl2di

þ � � � þ

lðd�1Þdi

Þ ¼ ð1�ldc

Þ=ð1�lÞ, we getPC�1

i ¼ 0 logð1þldi

þl2di

þ � � �

þlðd�1Þdi

Þ ¼ logð1�ldc

Þ�logð1�lÞ. Based on the result in Mitzen-

macher (1997), we have (3.2). &

Adding a constraint on node capacity makes it difficult to findthe closed-form solution to (3.1) with parameters l and d. Toanalyze the dynamic behaviors of the P2P system with homo-geneous and bounded nodal capacity, we conduct simulations totrace the changes of state variables in an example system. In thesystem, we have four peer nodes, each of which can queue up tosix queries at a time, and the arrival rate of queries is l¼ 0:99.The initial system is empty and queries come and leave the P2Psystem following the model described at the beginning of thissection. Figs. 1(a)–(c) depict the dynamics of the node queuelength, when one, two and three choices are made at random inprobing. It is clear that as d increases the number of nodes withthe longest queues decreases. This is because incoming queriesare more evenly distributed among nodes for greater d. Fig. 1(d)shows the value of C=ðlogðC=2ÞlogdÞ � TdðlÞ=logT1ðlÞ convergesclosely to 1 as time goes on. There is a small deviation from 1,because the arrival rate of queries l is set to 0.99 in order tosimulate l-1�. The system is stable and we have differenttrajectory for different value of d. From this figure, we canmeasure the expected time that a query stays in the P2P system,i.e. Td.

We apply Kurtz’s theorem (Shwartz and Weiss, 1995) to ourrandomized probing model (dZ2) to obtain bounds on the

maximum load:

Theorem 4. For any fixed time interval I, the length of the longest

queue in an initially empty P 2P system with homogeneous

and bounded node capacity over an interval [0,I] is

ðC=ðlogðC=2ÞlogdÞÞlog logNþOð1Þ with high probability, where C is

the node capacity and the form of the O(1) term depends on I and l.

Hence in comparing a system where queries have one choicewith that having dZ2 choices, we can see that the second oneproduces an exponential improvement with regard to both theexpected time of a query in the system and the maximumobserved load of nodes, for sufficiently large N.

3.1.2. Heterogeneous and bounded case

In Section 3.1.1, we extended the supermarket model toanalyze the effect of d-way random probing in balancing load inP2P systems with homogeneous and bounded nodal capacity.However, in practical P2P networks, participant nodes generallyhave different configurations. To tackle this nodal heterogeneity,we extend the preceding model to analyze behaviors of peernodes with different capacities in face of randomized probing tobalance load.

Here, we still consider P2P systems with static composition.Let us assume a system has N peer nodes to process queries. Theircorrespondent capacities are fc1,c2, . . . ,cNg, which are positive andfinite. We assume ci’s take nonuniform values, otherwise we cananalyze the system by applying the model presented in Section3.1.1. Next, we will try to model an investigate behavior of the P2Psystem with heterogeneous and bounded nodal capacity byutilizing results derived in the preceding section.

Let c* denote the maximum values in the sequence fcigNi ¼ 1.

Then, we calculate the residue capacity as fc��cigNi ¼ 1. We treat

these residue capacity as the initial load of their correspondingnodes. Thus, value of the state variables si for 0r irc� at timet¼ 0 equals to sið0Þ ¼ jfcijcirc��igj=N, i.e. the fraction of nodesbearing initial load (residue capacity) no less than i. When thesystem runs and queries come/leave, the area of residue capacityis reserved and load changes in the rest area within c*. With thistransformation, peer nodes have homogeneous capacity as c* sothat we can model the dynamic system by (3.1). The statevariables satisfy the following equations:

_si ¼ lðsdi�1�sd

i Þ�ðsi�siþ1Þ, ioC,

_sC ¼ lðsdC�1�sd

C�sC ,

sið0Þ ¼jfcijcirc��igj

N, irC,

siðtÞZsið0Þ:

8>>>>>><>>>>>>:

ð3:4Þ

For example, in a small-scale P2P network having four nodes, theycan accommodate at most 3, 3, 4 and 6 queries at a time, respec-tively. Thus, c� ¼ 6 and their residue capacities are c ¼ f3,3,2,0g,which determines the initial load of the corresponding nodesfsið0Þg

6i ¼ 0 ¼ f1,0:75,0:75,0:5,0,0,0g. The system dynamics can be

modeled by (3.4) and its solution describes the steady states ofthe P2P system.

Eqs. (3.4) are established by exerting constraints on the initialvalues and the range of state variables to (3.1). Their equilibriumstates have certain relations.

Corollary 5. Let ~T dðlÞ denote the expected time a query spends in

the system (3.4). Then, ~T dðlÞ is bounded by

TdðlÞr ~T dðlÞrTdðlÞþS,

where TdðlÞ is the expected time a query spends in the homogeneous

and bounded-capacity system (3.1) and S is a constant which equals

toPC

i ¼ 1 sið0Þ.

0 50 100 150 200 250 300 350 4000

0.10.20.30.40.50.60.70.80.9

1

Time

Frac

tion

of n

odes

with

que

ue le

ngth

i

Frac

tion

of n

odes

with

que

ue le

ngth

i

s6

s5

s4

s3

s2

s1

0 50 100 150 200 250 300 350 4000

0.10.20.30.40.50.60.70.80.9

1

Time

s6

s5

s4

s3s2s1

0 50 100 150 200 250 300 350 4000

0.10.20.30.40.50.60.70.80.9

1

Time

Frac

tion

of n

odes

with

que

ue le

ngth

i

s6

s5

s4

s3s2s1

0 50 100 150 200 250 300 350 4000.30.40.50.60.70.80.9

11.1

Timed-

way

resp

onse

tim

e / l

og(1

-way

resp

onse

tim

e)

d = 2d = 3

d = 4

Fig. 1. Dynamics of query response time and node queue length in a simulated P2P network with homogeneous and bounded-capacity nodes. (a) State dynamics when

d¼1. (b) State dynamics when d¼2. (c) State dynamics when d¼3. (d) aTdðlÞ=logT1ðlÞ changes with time.

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261256

Proof. In Section 3.1.1, we prove that (3.1) converges doublyexponentially to its equilibrium state fpig. We now derive thesolution to (3.4) based on fpig. Let fmig

Ci ¼ 1 ¼ fmaxðpi,sið0ÞÞg

Ci ¼ 1.

We calculate the remains of (lmdi�1�mi) for 1r irC. Let

r¼minfðlmdi�1�miÞ

Ci ¼ 1g. Then, we obtain the solution ð ~p iÞ by

iteratively computing ~p i ¼ l ~pdi�1�mi for 1r irC, starting with

~p0 ¼ 1.

If fpig is no less than the value of the initial state fsið0Þg in (3.4),

then fpig is also the steady state of (3.4).~T dðlÞ ¼

PCi ¼ 1 ið ~pd

i�1� ~pdi Þ ¼

PCi ¼ 1 ~p i. Therefore, ~T dðlÞZTdðlÞ. If

fpig is less than fsið0Þg, we have ~p irpiþsið0Þ. The expected time

a query spends in the system satisfies ~T dðlÞrTdðlÞþS, where S is

a constant which equals toPC

i ¼ 1 sið0Þ. &

Based on Corollary 5, we apply Kurtz’s theorem to derive theupper bound of the length of queues in the P2P system withheterogeneous and bounded nodal capacity.

Theorem 6. For any fixed time interval I, the length of the longest

queue in an initially empty P 2P system with heterogeneous and

bounded nodal capacity for dZ2 over the interval [0, I] is

ðc�=logðc�=2Þlog dÞlog logNþOð1Þ with high probability, where c* is

the upper bound of nodal capacity, and the O(1) term depends on c*,si(0), I and l.

The result of Theorem 6 is reduced to the one in Theorem 4when the system is homogeneous. This theorem indicates that themaximum load of peer nodes is affected by the distributions ofnodal capacity. However, the power of 2-way randomized probingis still valid in P2P networks with heterogeneous and boundednodal capacity.

3.2. Node dynamics

The composition of a P2P network is dynamic. Compute nodesjoin and leave the P2P system in its lifetime. Guha et al. (2006)observed that only 30–40% supernodes were online at any giventime in a P2P network. Nodal churn must be considered when weanalyze the behaviors of randomized probing to balance load

among peer nodes. In this section, we model the dynamiccomposition of peer nodes in a P2P network by using a randomvariable. Then we investigate the impact of nodal churn to theexpect time a queue spends in the system and the longest lengthof queues among peer nodes. To make our analysis tractable, wefirst discuss randomized probing in a dynamic P2P network wherenodes have infinite capacity, as in Section 3.2.1. Then, peer nodeswith bounded capacities are investigated in Section 3.2.2.

3.2.1. Dynamic nodes with unbounded capacity

In Section 3.1, we analyze the properties of randomizedprobing in static P2P systems by focusing on the factor of nodalcapacity. In a dynamic P2P system with nodal churn, nodecomposition is not fixed any more. We use a random variable n tocharacterize the number of peer nodes that changes with time.Then, variable mi(t), the number of nodes whose loads are at leasti, is also random. Their ratio mi(t)/n denoted by si(t) still describesthe fraction of peer nodes bearing loads that are at least i.

Existing research work (Yao et al., 2006; Stutzbach and Rejaie,2006; Castro et al., 2005; Krishnamurthy et al., 2005; Rhea et al.,2004) on analyzing churn in P2P systems found that the arrival/departure processes of peer nodes can be modeled by a Poissondistribution when the system size becomes sufficiently large.Thus, we assume new nodes join the current P2P network in aPoisson distribution with rate lin and existing nodes leave thesystem in a Poisson distribution with rate lout relative to the nodepopulation, lin,lout o1. When a peer node leaves, its original loadwill be removed from the P2P system. We assume the probabilitywith which a node leaves the system is uniformly distributedamong the existing nodes. As a result, the number of nodes join/leave the P2P system is Dn¼ ðlin�loutÞnDt in a small time intervalDt. A new node can service coming queries when they aredispatched to it.

To characterize the system dynamics, we look into the changeof random variable mi, the number of nodes that have load at leasti. Its value changes when a query is dispatched to a node queuingi�1 queries, or a query is serviced and removed from a node

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261 257

having i queries or such a node leaves the system. Therefore,

Dmi

Dt¼ ln

mi�1

n

� �d

�mi

n

� �d� �

�nmi

n�

miþ1

n

� ��loutn

mi�miþ1

n:

Thus, the influence of nodal churn on system behavior isincorporated in the value of random variable mi. Becausesi ¼mi=n, we have _si ¼ ð _min�mi _nÞ=n2. We use fsig

1i ¼ 0 and n as

state variables and the state equations characterizing systemdynamics are as follows.

_si ¼ lðsdi�1�sd

i Þ�ð1þlinÞsiþð1þloutÞsiþ1, iZ1,

_n ¼ ðlin�loutÞn,

s0 ¼ 1,

8><>: ð3:5Þ

with the initial condition sið0Þ ¼ 0, 1r irn and nð0Þ ¼N, where N

is the number of nodes in the initial system.The population of the P2P system depends on the values of lin

and lout . If lin4lout , n tends to increase. On the other hand, whenn increases, more queries will arrive at the system. A similarsituation happens when linolout . Thus, although nodes join/leavethe system in runtime, the state variables fsig

1i ¼ 0 can converge to

steady state by adjusting the volume of queries.

Corollary 7. Let TdðlÞ denote the expected time a query spends in the

system (3.5). Then, Tdð~lÞ is bounded by

0rTdð~lÞr

1þlin

1þloutld�1T̂ dðlÞ,

where ~l ¼ fl,lin,loutg and T̂ dðlÞ is the expected time a query spends

in the homogeneous and infinite-capacity system (Mitzenmacher,2001).

Proof. Let _si ¼ 0 in (3.5) for iZ1, and we getsiþ1 ¼ ð1=ð1þloutÞÞ½ð1þlinÞsi�lsd

i�1þlsdi �.

Tdð~lÞ ¼

P1i ¼ 1 iðpd

i�1�pdi Þ ¼

P1i ¼ 0 pd

i . By using induction, we canprove the expression in the corollary. &

Figs. 2(a)–(c) depict dynamics of the nodal queue length whenone, two and three choices are made by random in node search:lin ¼ 0:2, lout ¼ 0:1 and l¼ 0:99. It is clear that as d increases the

0 10 20 30 40 50 60 70 80 90 1000

0.10.20.30.40.50.60.70.80.9

1

Time

Frac

tion

of n

odes

with

que

ue le

ngth

i

s5s6

s4

s3

s2

s1

0 10 20 30 40 50 60 70 80 90 1000

0.10.20.30.40.50.60.70.80.9

1

Time

Frac

tion

of n

odes

with

que

ue le

ngth

i s1

s2

s3s4-s6

Fig. 2. Dynamics of query response time and nodal queue length in a simulated P2P netw

State dynamics when d¼2. (c) State dynamics when d¼3. (d) aTdð~lÞ=logT1ð

~lÞ changes

number of nodes with the longest queues decreases, becauseincoming queries are more balanced distributed among nodes forlarger d. Fig. 2(d) presents the values of ð1þloutÞ=ð1þlinÞlogd �

TdðlÞ=logT1ðlÞ for as l-1�. We can see they converge closely to 1as time goes on. The small deviation from 1 is because the arrivalrate of queries l is set to 0.99 as to simulate l-1�.

We apply Kurtz’s theorem (Shwartz and Weiss, 1995) to ourrandomized probing model in P2P network to obtain bounds onthe maximum load:

Theorem 8. In an initially empty P2P system where nodes’ arrival/departure follows a Poisson distribution with rate linn and loutn andnodes have infinite capacity, for any fixed I, the length of the longest

queue in the system for dZ2 over the interval [0, I] is OðloglognÞ

with high probability.

3.2.2. Dynamic nodes with bounded capacity

A more general case is a P2P system that peer nodes arrive/depart at runtime and the participant nodes have heterogeneousand bounded capacities. In this section, we construct the stateequations to describe the system dynamics and derive the boundson length of the longest queue.

Let c1,c2, . . . ,cn denote the capacities of peer nodes in the system.The number of nodes n is a random variable. We assume fcig

ni ¼ 1

follows a Pareto distribution, with a shape parameter kc. The numberof nodes bearing load at least i, mi(t), follows the equations:

_mi ¼ lnmi�1

n

� �d

�mi

n

� �d� �

�nmi

n�

miþ1

n

h iþlinn � Prfc��cZ ig�loutn

mi�miþ1

n, ioc�,

_mc� ¼ lnmc��1

n

� �d

�mc�

n

� �d� �

�nmc�

n�loutn

mc�

n, i¼ c�,

_n ¼ ðlin�loutÞn,

8>>>>>>>>>><>>>>>>>>>>:where c* is a sufficiently large value that is greater than any possiblevalue of node capacity. According to the CDF of Pareto distribu-tion, Prfc��cZ ig ¼ Prfcrc��ig ¼ 1�ððc��iÞ=cminÞ

�kc , where cmin is aminimum capacity.

0 10 20 30 40 50 60 70 80 90 1000

0.10.20.30.40.50.60.70.80.9

1

Time

Frac

tion

of n

odes

with

que

ue le

ngth

i

s1

s2

s3

s4 s5-s6

0 10 20 30 40 50 60 70 80 90 1000

0.5

1

1.5

2

2.5

Time

d-w

ay re

spon

se ti

me

/ log

(1-w

ay re

spon

se ti

me)

d = 2 d =3

d = 4

ork with churn and unbounded-capacity nodes. (a) State dynamics when d¼1. (b)

with time.

Table 1Experiment settings and algorithm parameters.

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261258

By applying _si ¼ ð _min�mi _nÞ=n2, we construct the followingstate equations:

_si ¼ lðsdi�1�sd

i Þ�ð1þlinÞsiþð1þloutÞsiþ1

þlin 1�c��i

cmin

� ��kc" #

, irc��cmin,

_si ¼ lðsdi�1�sd

i Þ�ð1þlinÞsiþð1þloutÞsiþ1, c��cmino ioc�,

_sc� ¼ lðsdc��1�sd

c� Þ�ð1þlinÞsc� , i¼ c�,

_n ¼ ðlin�loutÞn,

sið0Þ ¼jfcijcirc��igj

N, irc�,

siðtÞ Z sið0Þ

8>>>>>>>>>>>>>>>>><>>>>>>>>>>>>>>>>>:

ð3:6Þ

with initial condition s0ð0Þ ¼ 1,sið0Þ,1r irnð0Þ set according tothe server configuration at t¼0.

It is difficult to calculate closed form solutions to (3.6).However, it is numerically solvable. Figs. 3(a)–(c) show thedynamics of state variables fsig in an example system. The systemconsists of 10 nodes at t¼0. Their capacity are within the set of{6,7,y,10}, and for each value in the range there are two nodeshaving that capacity. lin, lout and l are 0.2, 0.1 and 0.99, respec-tively. The capacity of newly joined nodes follows a Paretodistribution with shape parameter kc¼2 and the minimumcapacity cmin¼5. The figures show the system evolves to steadystates as time goes on. We can calculate Tdð

~lÞ based on values ofthese steady states.

Environment parameter Default value

Object arrival location Uniform over Id space

Uniform over Id space 4096

Node capacity Bounded Pareto: shape 2

Lower bound:2500, upper bound: 2500*10

Number of items 20 480

Existing item load Bounded Pareto: shape: 2

Lower bound: mean item actual load/2

Upper bound: mean item actual load/2*10

4. Experimental results

To quantify the performance of random choices schemes inreal P2P systems, we conducted simulations on Cycloid (Shenet al., 2006) P2P network. Cycloid is a constant-degree DHT basedon the network topology of cube-connected-cycle. We designedand implemented a simulator in Java for evaluation of the load

0 10 20 30 40 50 60 70 80 90 1000

0.10.20.30.40.50.60.70.80.9

1

Time

Frac

tion

of n

odes

with

que

ue le

ngth

i

Frac

tion

of n

odes

with

que

ue le

ngth

i

s1

s2

s3

s4

s5s6s7s8s9s10

0 10 20 30

0.10.20.30.40.50.60.70.80.9

1

Fig. 3. Dynamics of query response time in a simulated P2P network with churn and

dynamics when d¼2. (c) State dynamics when d¼3.

balancing algorithms on Cycloid. Table 1 lists the parameters ofthe simulation and their default values.

GT-ITM (transit-stub and tiers) (Zegura et al., 1996) is anetwork topology generator, widely used for the construction ofoverlay networks. We used GT-ITM to generate transit-stubtopologies for Cycloid, and get physical hop distance for eachpair of Cycloid nodes. We selects the routing table entriespointing to the physically nearest among all nodes with nodeIDin the desired portion of the ID space.

We use landmark clustering and Hilbert number (Xu et al., 2003)to cluster Cycloid nodes. Landmark clustering is based on theintuition that close nodes are likely to have similar distances to afew landmark nodes. Hilbert number can convert d dimensionallandmark vector of each node to one dimensional index while stillpreserve the closeness of nodes. We selected 15 nodes as landmarknodes to generate the landmark vector and a Hilbert number foreach node cubic ID. Because the nodes in a stub domain have close(or even same) Hilbert numbers, their cubic IDs are also close to eachother. As a result, physically close nodes are close to each other inthe DHT’s ID space, and nodes in one cycle are physically close toeach other. For example, assume nodes i and j are very close to eachother in physical locations but far away from node m. Nodes i and j

will get approximately equivalent landmark vectors, which aredifferent from m’s. As a result, nodes i and j would get the same

Frac

tion

of n

odes

with

que

ue le

ngth

i

0 10 20 30 40 50 60 70 80 90 1000

0.10.20.30.40.50.60.70.80.9

1

Time

s1

s2

s3

s4

s5s6-s10

40 50 60 70 80 90 100

Time

s1

s2

s3

s4

s5-s10

heterogeneous, bounded-capacity nodes. (a) State dynamics when d¼1. (b) State

0

1000

2000

3000

4000

5000

6000

7000

8000

0.5System Utilization

Tota

l Prim

ary

Nod

es P

robi

ng T

ime

LAR1LAR2LAR4LAR6

0.6 0.7 0.8 0.9 1

Fig. 4. Total node probing time.

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261 259

cubic IDs and be assigned to the circle different from m’s. In thelandmark approach, for each topology, we choose landmarks atrandom with the only condition that the landmarks are separatedfrom each other by four hops.

Our experiments are built on two transit-stub topologies:‘‘ts5k-large’’ and ‘‘ts5k-small’’ with approximately 5000 nodeseach. In the topologies, nodes are organized into logical domains.We classify the domains into two types: transit domains and stubdomains. Nodes in a stub domain are typically an endpoint in anetwork flow; nodes in transit domains are typically intermediatein a network flow. ‘‘ts5k-large’’ has five transit domains, threetransit nodes per transit domain, five stub domains attached toeach transit node, and 60 nodes in each stub domain on average.‘‘ts5k-small’’ has 120 transit domains, five transit nodes pertransit domain, four stub domains attached to each transit node,and two nodes in each stub domain on average. ‘‘ts5k-large’’ has alarger backbone and sparser edge network (stub) than ‘‘ts5k-small’’. ‘‘ts5k-large’’ is used to represent a situation in whichCycloid overlay consists of nodes from several big stub domains,while ‘‘ts5k-small’’ represents a situation in which Cycloid overlayconsists of nodes scattered in the entire Internet and only fewnodes from the same edge network join the overlay. To accountfor the fact that inter-domain routes have higher latency, eachinter-domain hop counts as 3 hops of units of latency whileeach intra-domain hop counts as 1 hop of unit of latency. We runeach trial of the simulation for 20T simulated seconds, where T isa parameterized load balancing period, and its default value is setto 60 s in our test. The item and node join/departure rates aremodeled by Poisson processes. The default rate of item join/departure rate is 0.4; that is, there are one item join and one itemdeparture every 2.5 s. We range node inter-arrival time from 10 to90 s, with 10 s increment in each step.

To balance load in a P2P network, probes are sent to a numberof peer nodes. Among the responders, the one that has the leastload will be selected. We refer to this class of randomized loadbalancing algorithms as d-way probing, denoted by LARd,d¼1,2,y. We compare the performance of 1, 2, 4, and 6-wayrandom probe schemes in terms of node probing time and totalnumber of load rearrangements. Figs. 4 and 5 show that theprobing efficiency of LARd ðd42Þ is almost the same as LAR2, eventhough they need to probe more nodes than LAR2. For example, asthe system utilization reaches 85%, the probing time and numberof load arrangements by LAR2 are reduced by more than 38%and 35%, respectively, compared with those by LAR1, while theperformance is almost the same as that by LAR3. The experimental

results are consistent with the theorems that we derive inSections 3.1 and 3.2 on the performance of randomized loadbalancing algorithms in P2P systems. A two-way probing methodleads to an exponential improvement over one-way probing, buta d-way ðd42Þ probing leads to much less substantial additionalimprovement.

5. Related work

In most early DHT structures (Stoica et al., 2003; Manku et al.,2003; Malkhi et al., 2002), each node chooses at random a point inthe hash space, typically, the unit interval [0, 1), and becomesassociated with the points of the hash space closest to the selectedpoint. Assuming random node departures, this scheme guaranteesthat the ratio of largest to average node segment size is O(log n),with high probability (Stoica et al., 2003). Virtual server approachhas been used to mitigate imbalance of key assignment betweennodes. It was proposed that each real server works as log n virtualservers, thus greatly decreasing the probability that some serverwill get a large part of the ring. Some extensions of this methodwere proposed in Rao et al. (2003) and Surana et al. (2006), wheremore schemes based on virtual servers were introduced andexperimentally evaluated. However, these schemes assume thatnodes are homogeneous, objects have the same size, and objectIDs are uniformly distributed.

CFS (Dabek et al., 2001) accounts for node heterogeneity byallocating to each node some number of virtual servers propor-tional to the node capacity. In addition, CFS proposes a simplesolution to shed the load from an overloaded node by having theoverloaded node remove some of its virtual servers. However, thisscheme may result in thrashing as removing some virtual serversfrom an overloaded node may result in another node becomingoverloaded.

Byers et al. (2003) proposed the use randomized search toachieve better load balance. Each object is hashed to dZ2different IDs, and is placed in the least loaded node of the nodesresponsible for those IDs. The other nodes are given a redirectionpointer to the selected node so that searching is not slowedsignificantly. For homogeneous nodes and objects and a staticsystem, picking d¼2 achieves a load balance within a log log n

factor of optimal. However, this scheme was not analyzed orsimulated for the case of heterogeneous node capacities and nodechurn, which are defining characteristics of P2P networks. Theparadigm of multiple random choices was also used in Shen and

0

1000

2000

3000

4000

5000

6000

7000

0.5System Utilization

Tota

l Num

ber o

f Loa

d R

earr

ange

men

ts

LAR1LAR2LAR4LAR6

0.6 0.7 0.8 0.9 1

Fig. 5. Total number of load rearrangements.

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261260

Xu (2006, 2007), Kenthapadi and Manku (2005), and Giakkoupisand Hadzilacos (2005). Several peer nodes are probed before storea file to or dispatch a user query to the least loaded one.

Besides applying randomized probing algorithms to balanceload in P2P networks, researchers also analyzed the character-istics of random choices theoretically. The main techniques usedto analyze random choice problems are layered induction, witnesstrees, and fluid limits via differential equations. The layered

induction technique pioneered by Azar et al. (2000). The randomchoice problem was modeled by balls-and-bins. It bounded themaximum load by bounding the number of bins with k or moreballs via induction on k. The layered induction approach providesnearly tight results. An alternative technique for handling theproblem called the witness tree method (Cole et al., 1998). The keyidea of this approach is to show that if a ‘‘bad event’’ occurs, i.e.if some node is heavily loaded, one can extract the history of theprocess a suitable tree of events called the witness tree.The probability of the bad event can then be bounded by theprobability of occurrence of a witness tree. Generally, witness treearguments involves the most complexity, and they have proved tobe the most challenging in terms of obtaining tight results. Thethird technique studies algorithms that use random choicesparadigm via fluid limit models (Mitzenmacher, 2001; Byerset al., 2004). The system dynamics can be described by a familyof differential equations. This approach is simple and flexible.When the system dynamics can be modeled by this method, thedifferential equations generally yield accurate numerical result-s.However, these theory work analyzed a system where computenodes have homogeneous and infinite capacities. Moreover, nodechurn, a defining characteristic of P2P systems, is not modeled bythese approaches. In this paper, we analyze the dynamic behaviorof random choice paradigm in general P2P systems, where peernodes join/leave at runtime and they have heterogeneous andbounded capacities.

To develop resilient systems, Fu and Xu (2010, 2007a, 2007b)analyzed and modeled the failure correlation in large-scalecompute cluster and grid systems. They also proposed failure-aware resource management mechanisms by using reconfigurabledistributed virtual machines in networked computer environ-ments (Fu, 2009, 2010a, 2010b).

6. Conclusions

In this paper, we model the randomized probing in generalpeer-to-peer systems. Theoretical analysis shows that two-wayrandom probing is asymptotically superior to the one-way choice

approach. However, by increasing the number of choices further,efficiency of the search algorithm does not improve significantly.Our random probing model is general in that the influence bynodal heterogeneity, capacity distribution and churn on searchefficiency is investigated. The random approach is less sensitive tothe node churn and heterogeneity in terms of the number ofprobes conducted before finding suitable nodes and the averageresponse time of queries. Simulation and experiment resultsconfirm our analysis. It is difficult to calculate the closed formsolutions to state equations of the most general case. However,we can find the steady states numerically. For completeness intheory, we include the analysis of P2P systems consistingheterogeneous and bounded-capacity nodes with churn andsimulation results. In this paper, we design and analyze therandom probing algorithms within the context of P2P networks.However, our results have wide applicability and are of interestbeyond the specific applications.

Although we analyze the performance of randomized probingin the general P2P environments, we still introduce somesimplifying assumptions to make the problem tractable. Onesuch assumption is that each peer node can service every query.However, in practice certain queries or requests may only beserviced by those nodes that have the required resources. Toaddress this situation, we need to extend the composition modelof a P2P network in a way that each query maps to a subset ofnodes that can service it, and the changes of peer nodes’ queuelength will be quantified by distinguishing these different sets.However, this will make the analysis quite difficult. Anotherassumption is that the service time for a query is exponentiallydistributed with mean 1. In reality, different queries may requiredifferent amount of work and the processing power of peer nodesmay vary. As a result, the service time follows a more complicatedmodel. Although we do not discuss this case in our paper, ourmodel can be extended to formulate the case by introducing otherdistributions of service time to the state equations of the systemand numerically analyze the state dynamics. In this paper, weevaluate the performance of LAR, a generic load balancingalgorithm based on d-way probing. We also plan to conduct moreexperiments to compare the performance of several other loadbalancing algorithms in addition to LAR.

Acknowledgments

This research was supported in part by US NSF Grant CNS-0915396 and LANL Grant IAS-1103. The authors thankDr. Mohammed Atiquzzaman and anonymous reviewers for their

S. Fu et al. / Journal of Network and Computer Applications 34 (2011) 252–261 261

constructive comments and suggestions. A preliminary versionof this paper was published in the Proceedings of the 22ndACM/IEEE International Parallel and Distributed ProcessingSymposium (IPDPS), 2008 (Fu et al., 2008).

References

Azar Y, Broder AZ, Karlin AR, Upfal E. Balanced allocations. SIAM Journal onComputing 2000;29(1):180–200 A preliminary version of this paper appearedin Proceedings of the 26th ACM symposium on Theory of computing (STOC),1994.

Bienkowski M, Korzeniowski M, auf der Heide FM. Dynamic load balancing indistributed hash tables. In: Proceedings of international workshop on peer-to-peer systems (IPTPS), February 2005.

Byers J, Considine J, Mitzenmacher M. Simple load balancing for distributed hashtables. In: Proceedings of the 2nd international workshop on peer-to-peersystems (IPTPS), 2003.

Byers JW, Considine J, Mitzenmacher M. Geometric generalizations of the power oftwo choices. In: Proceedings of the 16th ACM symposium on parallelism inalgorithms and architectures (SPAA), 2004.

Castro M, Costa M, Rowstron A. In: Proceedings of the 2nd symposium onnetworked system design and implementation (NSDI), 2005.

Cole R, Maggs BM, auf der Heide FM, Mitzenmacher M, Richa AW, Schrder K, et al.Randomized protocols for low-congestion circuit routing in multistageinterconnection networks. In: Proceedings of the 13th ACM symposium ontheory of computing (STOC), 1998.

Dabek F, Kaashoek MF, Karger D, Morris R, Stoica I. Wide-area cooperative storagewith cfs. In: Proceedings of the 18th ACM symposium on operating systemsprinciples (SOSP), 2001.

Fu S. Failure-aware construction and reconfiguration of distributed virtualmachines for high availability computing. In: Proceedings of IEEE/ACMinternational symposium on cluster computing and the grid (CCGrid), 2009.

Fu S. Dependability enhancement for coalition clusters with autonomic failuremanagement. In: Proceedings of IEEE international symposium on computersand communications (ISCC), 2010a.

Fu S. Failure-aware resource management for high-availability computing clusterswith distributed virtual machines. Journal of Parallel and DistributedComputing 2010b;70(4):384–93.

Fu S, Xu C-Z. Exploring event correlation for failure prediction in coalitions ofclusters. In: Proceedings of ACM/IEEE conference on supercomputing (SC),2007a.

Fu S, Xu C-Z. Quantifying temporal and spatial correlation of failure events forproactive management. In: Proceedings of IEEE international symposium onreliable distributed systems (SRDS), 2007b.

Fu S, Xu, C-Z. Quantifying event correlations for proactive failure management innetworked computing systems. Journal of Parallel and Distributed Computing2010, doi:10.1016/j.jpdc.2010.06.010.

Fu S, Xu C-Z, Shen H. Random choices for churn resilient load balancing in peer-to-peer networks. In: Proceedings of the 22nd ACM/IEEE international sympo-sium on parallel and distributed processing (IPDPS), 2008.

Giakkoupis G, Hadzilacos V. A scheme for load balancing in heterogenousdistributed hash tables. In: Proceedings of the 24th ACM symposium onprinciples of distributed computing (PODC), 2005.

Godfrey B, Stoica I. Heterogeneity and load balance in distributed hash tables. In:Proceedings of IEEE conference on computer communications (INFOCOM),2005.

Guha S, Daswani N, Jain R. An experimental study of the skype peer-to-peer voipsystem. In: Proceedings of 5th international workshop on peer-to-peersystems (IPTPS), 2006.

Kenthapadi K, Manku GS. Decentralized algorithms using both local and randomprobes for P2P load balancing. In: Proceedings of the 7th ACM symposium onparallelism in algorithms and architectures (SPAA), 2005.

Krishnamurthy S, El-Ansarh S, Aurell E, Haridi S. A statistical theory of chord underchurn. In: Proceedings of 4th international workshop on peer-to-peer systems(IPTPS), 2005.

Malkhi D, Naor M, Ratajczak D. Viceroy: a scalable and dynamic emulation of thebutterfly. In: Proceedings of the 21st ACM symposium on principles ofdistributed computing (PODC), 2002.

Manku G, Bawa M, Raghavan P. Symphony: distributed hashing in a small world.In: Proceedings of the 4th USENIX symposium on internet technologies andsystems (USITS), 2003.

Mitzenmacher M. On the analysis of randomized load balancing schemes. In:Proceedings of ACM symposium on parallel algorithms and architectures(SPAA), 1997.

Mitzenmacher M. The power of two choices in randomized load balancing. IEEETransactions on Parallel Distributed Systems 2001;12(10):1094–104.

Mitzenmacher M, Prabhakar B, Shah D. Load balancing with memory. In:Proceedings of the 43rd IEEE symposium on foundations of computer science(FOCS), 2002.

Rao A, Lakshminarayanan K, Surana S. Load balancing in structured P2P systems.In: Proceedings of the 2nd international workshop on peer-to-peer systems(IPTPS), 2003.

Ratnasamy S, Francis P, Handley M, Karp R, Shenker S. A scalable contentaddressable network. In: Proceedings of conference on applications, technol-ogies, architectures, and protocols for computer communications (SIGCOMM),2001.

Rhea S, Geels D, Roscoe T, Kubiatowicz J. Handling churn in a DHT. In: Proceedingsof USENIX annual technical conference, 2004.

Rowstron A, Druschel P. Pastry: scalable, distributed object location and routingfor large-scale peer-to-peer systems. In: Proceedings of the 18th IFIP/ACMinternational conference on distributed systems platforms (Middleware 2001),2001.

Shen H, Xu C-Z. Elastic routing table with provable performance for congestioncontrol in DHT networks. In: Proceedings of the 26th IEEE internationalconference on distributed computing systems (ICDCS), 2006.

Shen H, Xu C-Z. Locality-aware and churn-resilient load balancing algorithms instructured peer-to-peer networks. IEEE Transactions on Parallel and Distrib-uted Systems 2007;18(6):849–62.

Shen H, Xu C-Z, Chen G. Cycloid: a constant-degree and lookup-efficient P2Poverlay network. Performance Evaluation 2006;63(3):195–216.

Shwartz A, Weiss A. Large deviations for performance analysis: queues, commu-nication and computing. London, UK: Chapman and Hall; 1995.

Stoica I, Morris R, Karger D, Kaashoek F, Balakrishnan H. Chord: a scalable peer-to-peer lookup service for internet applications. IEEE/ACM Transactions onNetworking 2003;11(1):17–32.

Stutzbach D, Rejaie R. Understanding churn in peer-to-peer networks. In:Proceedings of ACM internet measurement conference (IMC), 2006.

Surana S, Godfrey B, Lakshminarayanan K, Karp R, Stoica I. Load balancing indynamic structured peer-to-peer systems. Performance Evaluation 2006;63(3):217–40.

Tewari S, Kleinrock L. Proportional replication in peer-to-peer networks. In:Proceedings of IEEE conference on computer communications (INFOCOM),2006.

Xu C-Z, Lau F. Load balancing in parallel computers: theory and practice. Springer-Verlag, Kluwer Academic; 1997.

Xu Z, Tang C, Zhang Z. Building topology-aware overlays using global soft-state. In:Proceedings of the 23rd IEEE international conference on distributedcomputing systems (ICDCS), 2003.

Yao Z, Leonard D, Wang X, Loguinov D. Modeling heterogeneous user churn andlocal resilience of unstructured P2P networks. In: Proceedings of 14thinternational conference on network protocols (ICNP), 2006.

Zegura EW, Calvert KL, Bhattacharjee S. How to model an internetwork. In:Proceedings of IEEE conference on computer communications (INFOCOM),1996.

Zhao BY, Huang L, Stribling J, Rhea SC, Joseph AD, Kubiatowicz J. Tapestry: aninfrastructure for fault-tolerant wide-area location and routing. Journal onSelected Areas in Communications 2004;12(1):41–53.

Zheng C, Shen G, Li S, Shenker S. Distributed segment tree: support of range queryand cover query over DHT. In: Proceedings of the 5th international workshopon peer-to-peer systems (IPTPS), 2006.

Zhu Y, Hu Y. Efficient, proximity-aware load balancing for DHT-based P2Psystems. IEEE Transactions on Parallel and Distributed Systems 2005;16(4):349–61.