Distributed Object Location in a Dynamic Network

Kirsten Hildrum, John D. Kubiatowicz, Satish Rao and Ben

Y. Zhao

DOLR(Decentralized Object Location and Routing)

Object

Object 1

Requirements

• Deterministic location

• Routing locality – routes should have low stretch

• Minimal storage and computational load

• Dynamic membership

System Stretch Storage Node Insertion

Metric

PRR, 1997 O(1) O(log n) -- Special

PRR + this O(1) O(log n) O(log2 n) Special

Tapestry, 2000 Low in practice

O(log n) -- General

Tapestry + this

Low in practice

O(log n) O(log2 n) General

Main Results

Additional Results

• Handle deletes

• Simultaneous insertions

• Algorithm with polylog stretch for general metrics.– Similar to Bourgain’s use of subsets to

embed a general metric into L2

Outline

• Background and previous work– PRR/Tapestry

• Insertion – Maintaining deterministic location– Low stretch: A nearest neighbor algorithm

• Related Work

• Conclusion

Neighbor MapFor “2175” (Octal)

Routing Levels1234

1xxx

2175

0xxx

3xxx

4xxx

5xxx

6xxx

7xxx

20xx

2175

22xx

23xx

24xx

25xx

26xx

27xx

210x

211x

212x

213x

214x

215x

216x

2175

2170

2171

2172

2173

Ø

2175

Ø

2177

Neighbor Table

1

NodeID2245

3

3

2

22

4

3

NodeID2175

NodeID2158

NodeID2421

NodeID2543

NodeID2163

NodeID2170

4

2

3

2

3

2

2

1

2

4

1

2

3

3

3

34

1

1

4 3

2

4

NodeID0xEF34

NodeID2170NodeID

2105

NodeID0732

NodeID0x2345

NodeID2173

NodeIDE324

NodeID2143

NodeID2172

NodeID3456

NodeID2056

NodeID2732

NodeID2163

NodeID0777

NodeID0775

NodeID2234

NodeID2161

NodeID2175

Basic Tapestry Mesh(from PRR97)

Use of Mesh for Object Location

Surrogate Routing

• Neighbor table will have holes

• Must be able to find unique root for every object

• Our solution: try next highest. – For 2176, try 2177– For 213? Try 214?

• Need: all Ø really are Ø -> root unique.

Neighbor MapFor “2175” (Octal)

Routing Levels1234

1xxx

2175

0xxx

3xxx

4xxx

5xxx

6xxx

7xxx

20xx

2175

22xx

23xx

24xx

25xx

26xx

27xx

210x

211x

212x

Ø

214x

215x

216x

2175

2170

2171

2172

2173

Ø

2175

Ø

2177

Outline Again

Neighbor MapFor “2175” (Octal)

Routing Levels1234

1xxx

2175

0xxx

3xxx

4xxx

5xxx

6xxx

7xxx

20xx

2175

22xx

23xx

24xx

25xx

26xx

27xx

210x

211x

212x

213x

214x

215x

216x

2175

2170

2171

2172

2173

Ø

2175

Ø

2177

Insertion

• Build own neighbor table– Want: Closest for

each entry (note: cannot copy from nearby node)

• Has to add self to other tables– Fill in holes– May replace further

node

1

Finding the Closest Node

• Building table means finding nearest neighbor for each entry.

• Our algorithm: find nearest level-i node for every i in process of finding overall nearest

• Easily extend to find other entries

Network Assumption

• Nearest neighbor is hard in general metric

• Assume the following:– Ball of radius 2r contains only a factor of c

more nodes than ball of radius r.– Also, b > c2

– [Both assumed by PRR]

• Start knowing one node; allow distance queries

Algorithm Idea

• Call a node a level i node if it matches the new node in i digits.

• The whole network is contained in forest of trees rooted at highest possible imax.

• Let list[imax] contain the root of all trees. Then, starting at imax, while i > 1– list[i-1] = getChildren(list[i])

• Certainly, list[i] contains level i neighbors.

NodeID0xEF34

NodeID0xEF31NodeID

0xEFBA

NodeID0x0921

NodeID0xE932

NodeID0xEF37

NodeID0xE324

NodeID0xEF97

NodeID0xEF32

NodeID0xFF37

NodeID0xE555

NodeID0xE530

NodeID0xEF44

NodeID0x0999

NodeID0x099F

NodeID0xE399

NodeID0xEF40

4

4

4

1

1

1

1

2

2

22

2

3

3

3

3

NodeID0xEF34

We Reach The Whole Network

The Real Algorithm

• Simplified version ALL nodes in the network.

• But far away nodes are not likely to have close descendents

Trim the list at each step.• New version: while i > 1

– List[i-1] = getChildren(list[i])– Trim(list[i-1])

How to Trim• Consider circle of

radius r with at least one level i node.

• Level-(i-1) node in little circle must must point to a level-i node in the big circle

• Want: list[i] had radius three times list[i-1] and list[i –1] contains one level i

<2r

r

Animation

new

True in Expectation

• Want: list[i] had radius three times list[i-1] and list[i –1] contains one level i

• Suppose list[i-1] has k elements and radius r – Expect ball of radius 4r to contain kc2/b– Ball of radius 3r contains less than k

nodes, so keeping k all along is enough.

• To work with high probability, k = O(log n)

Steps of Insertion•Find node with closest matching ID (surrogate) and get preliminary neighbor table

– If surrogate’s table is hole-free, so is this one.

•Find all nodes that need to put new node in routing table via multicast•Optimize neighbor table

– w.h.p. contacted nodes in building table only ones that need to update their own tables

•Need:

No fillable holes.

Keep objects reachable

• Need-to-know = a node with a hole in neighbor table filled by new node

• If 1234 is new node, and no 123s existed, must notify 12?? Nodes

• Acknowledged multicast to all matching nodes

Need-to-know nodes

Locates & Contacts all nodes with a given prefix

• Create a tree based on IDs as we go • Nodes send acks when all children reached• Starting node knows when all nodes reached

5434554340

The node then sends to any 5430?, any 5431?, any 5434?, etc. if possible

543??

5431? 5434?

Acknowledged Multicast Algorithm

Outline Again

Related Work• Many peer-to-peer systems

– Not locality-aware• Chord Stoica, Morris, Karger, Kaashoek, Balakrishnan • CAN Ratnasamy, Francis, Handley, Karp, Shenker • Pastry Rowstron and Druschel

– Polylog stretch, general metric• Awerbuch and Peleg 1991 (not dynamic)• Rajaraman, Richa, Vöcking, Vuppuluri 2001

– Constant stretch, special metric• Plaxton, Rajaraman, Richa 1997 (not dynamic)

• Nearest Neighbor– Karger and Ruhl’s 2002– Thorup and Zwick 2001– Clarkson 1997

Future Work

• Getting stretch is low in practical systems– Transit stub networks– Network structure inside stub

• Periodic optimization

• Better handling of inserts in presence of deletes

Extra/Backup Slides

System Node Insertion

Stretch Hops Metric

CAN, 2001 O(r) -- O(rn1/r) General

Chord, 2001 O(log2 n) -- O(log n) General

Pastry, 2001 O(log2 n) -- O(log n) General

Awerbuch Peleg, 1991

-- polylog polylog General

PRR, 1997 -- O(1) O(log n) Special

RRVV, 2001 polylog polylog polylog General

PRR + this O(log2 n) O(1) O(log n) Special

Tapestry O(log2 n) Low in practice

O(log n) General

Related Work

Maintaining Object Availability

• Object requests continue• New node: resends requests for object

it does not have via surrogate• Established node: always checks that

that it is final destination.– Sometimes-checking would require state.

• May get cycle for non-existent objects, but can be detected.

Body

• Finding the nearest neighbor– Can query for distance between self and

any given node.– Assume a low expantion property (low

dimentional)• Karger and Ruhl, May 2002. • Our algorithm not as good for general problem,

but solves our particular problem better– No additional storage needed– Finds all levels of table

Picture

new

Introduction

• Object Location/DOLR slide• Related Work table

– Not much done with stretch

• Our results– Add insertion to PRR/Tapestry

• Deletes• Parallel inserts

– Solve nearest neighbor problem inprocess• Similar to Karger and Ruhl

– Result for general metric spaces