A Common API for Structured Peer-to-Peer Overlays

A Common API for Structured Peer-to-Peer OverlaysFrank Dabek, Ben Y. Zhao,Peter Druschel, Ion Stoica

OceanStore / Sahara Retreat [email protected] January 14, 2003

Structured Peer-to-Peer Overlay They are:

Scalable, self-organizing overlay networks Provide routing to location-independent names Examples: CAN, Chord, Pastry, Tapestry, …

Basic operation: Large sparse namespace N

(integers: 0–2128 or 0–2160) Nodes in overlay network have nodeIds N Given k N, a deterministic function maps k

to its root node (a live node in the network) route(msg, k) delivers msg to root(k)


Current Progress Lots of applications built on top

File systems, archival backup Application level multicast Routing for anonymity, attack resilience

But do we really understand them? What is the core functionality that applications leverage from

them? What are the strengths and weaknesses of each protocol? How

can they be exploited by applications? How can we build new protocols customized to our future

needs?


Our Goals Protocol comparison

Compare and contrast protocol semantics Identify basic commonalities Isolate and understand differences

Towards a common API Easily supportable by old and new protocols Enables application portability between protocols Enables common benchmarks Provides a framework for reusable components


Talk Outline

Motivation

DHTs and DOLRs

A Flexible Routing API

Usage Examples


Decomposing Functional Layers Distributed Hash Tables (DHT)

put(key, data), value = get(key) Hashtable layered across network Handles replication; distributes replicas randomly Routes queries towards replicas by name

Decentralized Object Location and Routing (DOLR) publish(objectId), route(msg, nodeId),

routeObj(msg, objectId, n) Application controls replication and placement Cache location pointers to replicas; queries quickly intersect

pointers and redirect to nearby replica(s)


DHT Illustrated


DOLR Illustrated


Architecture

Routing Mesh Tier 0

Tier 1

Tier 2CFS PAST SplitStream i3 OceanStore

Replication

Bayeux

MulticastCAN,Chord+DHash

DHT TapestryPastry+Scribe

DOLR


Talk Outline

Motivation

DHTs and DOLRs


Usage Examples


Flexible API for Routing Goal

Consistent API for leveraging routing mesh Flexible enough to build higher abstractions

Openness promotes new abstractions Allow competitive selection to determine right abstractions

Three main components Invoking routing functionality Accessing namespace mapping properties Open, flexible upcall interface


API (routing)Data types Key, nodeId = 160 bit integer Node = Address (IP + port #), nodeId Msg: application-specific msg of arbitrary size

Invoking routing functionality Route(key, msg, [node])

route message to node currently responsible for key Non-blocking, best effort – message may be lost or duplicated. node: transport address of the node last associated with key

(proposed first hop, optional)


API (namespace properties) nextHopSet = local_lookup(key, num, safe)

Returns a set of at most num nodes from the local routing table that are possible next hops towards the key.

Safe: whether choice of nodes is randomly chosen nodehandle[ ] = neighborSet(max_rank)

Returns unordered set of nodes as neighbors of the current node. Neighbor of rank i is responsible for keys on this node should all

neighbors of rank < i fail nodehandle[ ] = replicaSet(key, num)

Returns ordered set of up to num nodes on which replicas of the object with key key can be stored.

Result is subset of neighborSet plus local node boolean = range(node, rank, lkey, rkey)

Returns whether current node would be responsible for the range specified by lkey and rkey, should the previous rank-1 nodes fail.


Deliver(key, msg) Delivers an incoming message to the application. One application

per node. Demultiplexing done by including demux key in msg. Forward(&key, &msg, &nextHopNode)

Synchronous upcall invoked at each node along route On return, will forward msg to nextHopNode App may modify key, msg, nextHopNode, or terminate by setting

nextHopNode to NULL. Update(node, boolean joined)

Upcall invoked to inform app of a change in the local node’s neighborSet, either a new node joining or an old node leaving.

API (upcalls)

msg msgdeliver forward

msgRouting Layer Routing Layer

Application Application


Talk Outline

Motivation

DHTs and DOLRs


Usage Examples


DHT Implementation Interface

put (key, value) value = get (key)

Implementation (source S, root R) Put: route(key, [PUT,value,S], NULL)

Reply: route(NULL, [PUT-ACK,key], S) Get: route(key, [GET,S], NULL)

Reply: route(NULL, [value,R], S)


DOLR Implementation Interface

RouteNode(msg, nodeId) Publish(objectId) RouteObj(msg, objectId, n)

Implementation (server S, client C, object O) RouteNode: route(nodeId, msg, NULL) Publish: route(objectId, [“publish”,O,S], NULL)

Upcall: addLocal([O,S]) RouteObj: route(nodeId, [n,msg], NULL)

Upcall: serverSet[] = getLocal(O);if (|serverSet|<n), route(nodeId, [n-|serverSet|,msg], NULL)for first n entries in serverSet, route(serverSet[i], msg, NULL)


Conclusion Very much ongoing work

Feedback valuable and appreciated

Ongoing Work Implementations will support routing API Working towards higher level abstractions

Distributed Hash Table APIDOLR publish/route API

For more information, see IPTPS 2003

Thank you…


Backup Slides Follow…


Storage API: Overview linsert(key, value); value = lget(key);


Storage API linsert(key, value): store the tuple <key,

value> into local storage. If a tuple with key already exists, it is replaced. The insertion is atomic wrt to failures of the local node.

value = lget(key): retrieves the value associated with key from local storage. Returns null if no tuple with key exists.


To Do

Following slides contain functions that we haven’t decided on yet…


Basic DHT API: Overview insert(key, value, lease); value = get(key); release(key);

Upcalls: insertData(key, value, lease);


Basic DHT API

insert(key, value, lease): inserts the tuple <key, value> into the DHT. The tuple is guaranteed to be stored in the DHT only for “lease” time. “value” also includes the type of operations to be performed on insertion. Default operation types include REPLACE: replace value associated with the same

key APPEND: append value to the existing key UPCALL: generate an upcall to application before

inserting …


Basic DHT API value = get(key): retrieves the value

associated with key. Returns null if no tuple with key exists in the DHT.


Basic DHT API Release(key): releases any tuples with

key from the DHT. After this operations completes, tuples with key are no longer guaranteed to exist in the DHT.


Basic DHT API: Open questions Semantics?

Verification/Access control/multiple DHTs? Caching?Replication?

Should we have leases? It makes us dependent on secure time sync.


Replicating DHT API Insert(key, value, numReplicas); adds a

numReplicas argument to insert. Ensures resilience of the tuple to up to numReplicas-1 “simultaneous” node failures.

Open questions: consistency


Caching DHT API

Same as basic DHT API. Implementation uses dynamic caching to balance query load.


Resilient DHT API

Same as replicating DHT API. Implementation uses dynamic caching to balance query load.


Publish API: Overview Publish(key, object); object = Lookup(key); Remove(key, object):


Publish API Publish(key, object): ensures that the locally

stored object can be located using the key. Multiple instances of the object may be published under the same key from different locations.

object = Lookup(key): locates the nearest instance of the object associated with key. Returns null if no such object exists.


Publish API Remove(key, object): after this operation

completes, the local instance of object can no longer be located using key.

Documents

A Common API for Structured Peer-to-Peer Overlays