Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems

Antony Bowstron & Peter Druschel

Presented by: Long Zhang

Outline

Introduction Design of Pastry

Node state & routing Pastry API Self-organization and adaptation Locality

Experimental Results Discussion

Introduction: What is Pastry?

It’s a scalable, distributed, decentralized object location and routing substrate

Serves as a general substrate for building P2P applications: SCRIBE, PAST,…etc.

Seeks to minimize distance messages travel

Pastry’s main capability

Pastry Node

Represented by 128-bit randomly chosen nodeId (Hash of IP or public key)

NodeId is in base 2b (b is a configuration parameter; b typical value 2 or 4)

Evenly distributed nodeIds along the circular namespace (0-2128 – 1 space).

Routes a message in O(log N) steps to destination N: size of network

Node state contains: Leaf Set ( L ) Routing table ( R ) Neighborhood Set ( M )

Design of Pastry: Node state

Leaf set: L Numerically closest nodes (L is a configuration parameter = 16, 32 typically )

Routing Table (Prefix-based)

Neighborhood Set: M physically closest nodes

Pastry node state (Leaf Set)

Serves as a fall back for routing table and contains: L/2 numerically closest and larger nodeIds L/2 numerically closest and smaller nodIds

Size of L is typically 2b or 2 x 2b

Nodes in L are numerically close (could be geographically diverse)

Pastry node state: Neighborhood set (M) Contains the IP addresses and nodeIds of

closest nodes according to proximity metric Size of |M| is typically 2b or 2x2b

Not used in routing, but instead for maintaining locality properties

Node state: Routing Table

Matrix of Log2b N rows and 2b – 1 columns (N is

the number of nodes in the network)

Entries in row n match the first n digits of current nodeId AND

Column number follows matched digits: Format: matched digits–column number–rest of ID

Log2b N populated on average

Node10233102 (2), (b = 2, l = 8)

0 1 2 302212102 22301203 31203203

11301233 12230203 1302102210031203 10132102 1032330210200230 10211302 102230210230322 10231000 1023212110233001 10233232

10233120

Routing(2):

If message with key D is within range of leaf set, forward to numerically closest leaf

Else forward to node that shares at least one more digit with D in its prefix than current nodeId

If no such node exists, forward to node that shares at least as many digits with D as current nodeId but numerically nearer than current nodeId

Routing Messages

(1) Node is in the leaf set

(2) Forward message to a closer node (Better match)

(3) Forward towards numericallyCloser node (not a better match)

D: Message KeyLi: ith closest NodeId in leaf setshl(A, B): Length of prefix shared by nodes A and BRi

j: (j, i)th entry of routing table

Routing Example:

Routing Performance:

(1) If key is within leaf set: target one hop away

(2) If key has to be routed: Number of nodes with longer prefix decreases by 2b

(3) Key is not covered by the leaf set (i.e., failed) With high probability, one more hop needed

Thus: Number of routing steps needed log 2b N

Pastry node join

X = new node, Z = numerically closest node, A = bootstrap (A is close in proximity space to X)

X sends a join message to A with target nodeId X A forwards to B C… Stops at Z, numerically closest to X’s nodeId

In process, A,B,…,Z send their state tables to X

Node Join

X’s neighborhood set (NS) = A’s NS X’s Leaf Set = Z’s leaf set X’s routing table is filled as follows:

X’s Row 0 = A’s row 0 (X0 = A0)

X’s Row 1 = B’s row 1 (X1 = B1)

…etc.

X sends its state to every node in its state tables ( Leaf set, neighborhood set, and routing table)

Node Join: Example

Node departure (2)

Invalid nodes in leaf set: detected by heartbeat monitor Repair by inserting node from another leaf’s LS

Heartbeat for neighborhood set (NS) Query all NS members for their NS tables, choose

replacement according to proximity metric Invalid routing entries detected when attempting

to route Query nodes in row for replacement entry, if failed Query successive rows until success

Node failure in routing table: example If node in red fails

Locality in Pastry

Based on proximity metric (i.e., No. of IP hops, geographic distance)

Proximity space is assumed to be Euclidean The route chosen for a message is likely to be “good“

with respect to the proximity metric We will discuss locality regarding:

Routing table locality Route locality Locating the nearest among k nodes

Locality in Routing tables

Invariant: “all routing table entries refer to a node that is near the present node, according to the proximity metric, among all live nodes with a prefix appropriate for the entry.”

We wish to maintain the invariant when adding new nodes.

X joins; A is close to X; X0 = A0, so locality holds in X’s routing table

X1 = B1. Entires in B1 (row 1 of X) are close to B, but are they necessarily close to X?

Locality in routing table

Entries of B1 are reasonable close to X Why? A is much closer to B than entry in B1 to B because every time

we choose from an exponentially decreasing set of nodes

To improve proximity approximation: X Queries nodes in routing table and neighborhood set for their

state Compares distances (from routing table entries) and update

route entries with closer nodes if found.

Route locality

At each routing step the message is moved closer to the destination in the: nodeId space (numerically closer nodes) proximity space: message travels the least possible distance

Given that: A message routed from A to B at a distance d cannot be

routed to a node with a distance of less than d from A. (follows from routing procedure)

Expected distance traveled increases exponentially Though shortest path is not guaranteed, we still get

a good route.

Locality among k nodes

In some Pastry-based applications, object is replicated on k nodes on its route (during insertion)

In prefix-base routing: goal is to reach any of k numerically closest nodes that has a copy of object

May miss nearby nodes with different prefix Use heuristic to determine when close to k nearest nodes

Based on density of nodeIds that store object; using local info Switch to numerically closest address

Arbitrary node failure

Node continues to be responsive, but behaves incorrectly or maliciously.

Repeated queries fail each time because they normally take the same route.

How to solve it? Use randomized routingThe choice among multiple nodes that satisfy

the routing criteria should be made randomly

Routing Performance

|L|=16 * b=4 * |M|=32 * 200,000 lookups

Pastry routing

Routing with failures

Pastry locality

Summary

Pastry is a generic P2P object location and routing substrate

Distributed, and scales well Used in developing applications like file

storage, global file sharing,...etc. Considers locality when routing messeges

Discussions

Security Compared with Chord and other P2P

overlay Other technique problems

Security

Pastry is an application-level routing and object location tool, which bear a security hazard, for example, if there is a malicious node who fake the routing table, I think it can fetch the useful information from others, or send the bad information which will impair the others. How to avoid it?

Security

They talk about using randomized paths to guard against malicious and/or failed nodes. While this may take care of an attacker who is mis-routing the information, what is done about attackers who try a Denial of Service attack by overloading the system? Also, is the integrity and confidentiality of messages guaranteed in case of a snooping attacker? Could it be possible to mount a man in the middle attack by routing a lot of packets through one's own node?

Compared with Chord and other P2P overlay 1. "Pastry" despite being a good design (in 2001)

failed to find real-world usage.

In fact CHORD, Pastry and other P2P designs from that era (~2001) didn't get popular.

Why do you think that "Pastry" and other first-generation DHT designs weren't successful (commercially or in terms of real-world usage)?

Compared with Chord and other P2P overlay Has this completely symmetric p2p

strategy "won" over hierarchical p2p, where there are supernodes which maintain information about a subset of nodes they are responsible for? (ex, FastTrack which Kazaa used). I mean "won" in terms of scalability and performance.

Compared with Chord and other p2p overlay Except the fact that pastry uses the prefix

of numerical representation to locate routing information, I think the idea is also somehow quite similar to Chord which we discussed in last class, is that correct? Which one is better? Can performance of Chord dominate Pastry?

Compared with Chord and other p2p overlay What large scale applications would want

to use a system like Pastry or Chord? If we had a large system that spanned many nodes, it seems like it would be worth the extra effort to optimize the mechanism to best suit the applications needs. (as opposed to a general purpose solution)

Compared with Chord and other p2p overlay In Pastry each node is required to keep

more information then Chord. It is hard to keep all these up to date even in Chord. How can it not be a large overhead in Pastry?

Compared with Chord and other p2p overlay There is no experiment about concurrent

joins. Will Pastry perform worse than Chord when concurrent joins happen, as it involves more complex updates even in the situation of single join?

Compared with Chord and other p2p overlay In the Pastry it tries to reach the node

closing down on the key positions in the NodeID. Would it be more suitable to consider the geographical proximity and route it ? The number of messages can still be minimize and is there any other system which consider geographical proximity in routing in P2P system ?

Other technique problems

In node arrival, X (the new node) will first contact A which is physically closest. What is benefit of first calling A rather than Z which is numerically closest to X ? (I do not see any GOOD reason why we should keep track of our neighbourhood set at all)

Other technique problems

In order to join de-centralized P2P networks we need to know the IP address of existing peer. (Even current generation KAD network uses bootstrap nodes .dat file which is distributed along with the application)

Is it possible to discover and connect to a P2P network without bootstrapping information?

Other technique problems

The authors mention using some sort of metric to determine the "distance" between two nodes, and also state that IP hop count may be a sensible choice of metric. Realistically, is this a useful metric? It seems that there are many better metrics (based on bandwidth, latency, etc), but they are harder to measure. Given the difficulty in measuring a truly useful metric of network distance, is it still valuable to minimize distance?

Other technique problems

The authors state that their implementation handled at least three thousand messages per second. In a real modern peer-to-peer network, is this a reasonable number? Or is the query rate much higher on a modern peer-to-peer application?

Other technique problems

The authors say that if the triangle inequality doesn't hold, the "locality properties of pastry routes may suffer", but then use an evenly distributed system of nodes to measure their performance metrics. Would it have been that hard to re run the experiments with a more clustered distribution of nodes in the plane, which is more realistic, and this way they could quantify how much the routes suffer.

Other technique problems

I don't understand why every row in the routing table has exponentially less nodes to chose from, and hence the nodes are exponentially further away distance wise. Given that the nodes are randomly distributed in the space, there should (probabilistically) be an equal number of nodes in the same area, that share a 1,2,3 or n bit prefix with the given node. The 2 shared prefixed nodes should be no further out than the 1 shared prefix ones, should they?

Other technique problems

Pastry uses an optimistic approach to controlling concurrent node arrivals and departures that the arrival/departure of a node affects only a small number of existing nodes in the system. Is this the case in practical p2p system?