*Towards A Common API for Structured Peer-to-Peer OverlaysFrank Dabek, Ben Y. Zhao,Peter Druschel, John Kubiatowicz, Ion Stoica

MIT, U. C. Berkeley, and Rice

* Conducted under the IRIS project (NSF)

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

State of the Art Lots and lots of peer to peer applications

Decentralized file systems, archival backup Group communication / coordination Routing layers for anonymity, attack resilience Scalable content distribution

Built on scalable, self-organizing overlays E.g. CAN, Chord, Pastry, Tapestry, Kademlia, etc…

Semantic differences Store/get data, locate objects, multicast / anycast How do these functional layers relate? What is the smallest common denominator?

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Some Abstractions Distributed Hash Tables (DHT)

Simple store and retrieve of values with a key Values can be of any type

Decentralized Object Location and Routing (DOLR) Decentralized directory service for endpoints/objects Route messages to nearest available endpoint

Multicast / Anycast (CAST) Scalable group communication Decentralized membership management

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Tier 1 Interfaces

DistributedHash Tables

(DHT)

DecentralizedObject Location / Routing (DOLR)

Multicast / Anycast(CAST)

put (key, data) publish (objectId) join (groupId)

remove (key)unpublish (objectId)

leave (groupId)

value = get (key)

sendToObj (msg, objectId, [n])

multicast (msg, gId)

anycast (msg, gId)

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Structured P2P Overlays

Key-based Routing Tier 0

Tier 1

Tier 2CFS PAST SplitStreami3 OceanStoreBayeux

CASTDHT DOLR

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

The Common Denominator Key-based Routing layer (Tier 0)

Large sparse Id space N(160 bits: 0 – 2160 represented as base b)

Nodes in overlay network have nodeIds N Given k N, overlay deterministically maps k

to its root node (a live node in the network) Goal: Standardize API at this layer Main routing call

route (key, msg, [node]) Route message to node currently responsible for key

Supplementary calls Flexible upcall interface for customized routing Accessing and managing the ID-space

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Deliver(key, msg) Delivers a message to application at the destination

Forward(&key, &msg, &nextHopNode) Synchronous upcall with normal next hop node Applications can override messages

Update(node, boolean joined) Upcall invoked to inform application of a node joining or leaving

the local node’s neighborSet

Flexible Routing via Upcalls

msgdeliver

KBR Layer

Application

msgforward

msgKBR Layer

Application

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

KBR API (managing ID space) Expose local routing table

nextHopSet = local_lookup (key, num, safe)

Query the ID space nodehandle[ ] = neighborSet (max_rank) nodehandle[ ] = replicaSet (key, num) boolean = range (node, rank, lkey, rkey)

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Caching DHT Illustrated

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Caching DHT Implementation Interface

put (key, value) value = get (key)

Implementation (source S, client C, root R) Put: route(key, [PUT,value,S])

Forward upcall: store valueDeliver upcall: store value

Get: route(key, [GET,C])Forward upcall: if cached, route(C, [value]), FINDeliver upcall: if found, route(C, [value])

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Ongoing Work What’s next

Better understanding of DOLR vs. CAST Decentralized endpoint management Policies in placement of indirection points

APIs and semantics for Tier 1: (DHT/DOLR/CAST) KBR API implementation in current protocols

See paper for additional details Implementation of Tier 1 interfaces on KBR KBR API support on selected P2P systems

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Backup Slides Follow…

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Our Goals Protocol comparison

Identify basic commonalities between systems Isolate and clarify interfaces by functionality

Towards a common API Easily supportable by old and new protocols Enable application portability between protocols Enable common benchmarks Provide a framework for reusable components

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Key-based Routing API Invoking routing functionality

Route (key, msg, [node]) Accessing routing layer

nextHopSet = local_lookup(key, num, safe) nodehandle[ ] = neighborSet(max_rank) nodehandle[ ] = replicaSet(key, num) boolean = range(node, rank, lkey, rkey)

Flexible upcalls: Delivery (key, msg) Forward (&key, &msg, &nextHopNode) Update (node, boolean joined)

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Observations Compare and contrast

Issues: locality, naming, caching, replication … Common: general algorithmic approach Contrast: instantiation, policy,

Revising abstraction definitions Pure abstraction vs. instantiated prototype

E.g. DHT abstraction vs. DHash Abstractions “colored” by initial application

E.g. Object location DOLR Endpoint Location and Routing Ongoing understanding of Tier 1 interfaces

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Flexible API for Routing Goal

Consistent API for leveraging routing mesh Flexible enough to build higher abstractions

Openness promotes new abstractions Allow competitive selection to determine right abstractions

Three main components Invoking routing functionality Accessing namespace mapping properties Open, flexible upcall interface

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

DOLR on Routing API

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

DOLR Implementation Endpoint E, client C, key K Publish: route(objectId, [“publish,” K, E])

Forward upcall: store ([K,E]) in local storage

sendToObj: route(nodeId, [n, msg])Forward upcall: e = Lookup (K) in local storage For (n |e|), route (ni, msg)

If (n > |e|), route (nodeId, [n - |e|, msg])

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Storage API: Overview linsert(key, value); value = lget(key);

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Storage API linsert(key, value): store the tuple <key,

value> into local storage. If a tuple with key already exists, it is replaced. The insertion is atomic wrt to failures of the local node.

value = lget(key): retrieves the value associated with key from local storage. Returns null if no tuple with key exists.

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Basic DHT API: Overview insert(key, value, lease); value = get(key); release(key);

Upcalls: insertData(key, value, lease);

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Basic DHT API

insert(key, value, lease): inserts the tuple <key, value> into the DHT. The tuple is guaranteed to be stored in the DHT only for “lease” time. “value” also includes the type of operations to be performed on insertion. Default operation types include REPLACE: replace value associated with the same

key APPEND: append value to the existing key UPCALL: generate an upcall to application before

inserting …

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Basic DHT API value = get(key): retrieves the value

associated with key. Returns null if no tuple with key exists in the DHT.

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Basic DHT API Release(key): releases any tuples with

key from the DHT. After this operations completes, tuples with key are no longer guaranteed to exist in the DHT.

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Basic DHT API: Open questions Semantics?

Verification/Access control/multiple DHTs? Caching?Replication?

Should we have leases? It makes us dependent on secure time sync.

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Replicating DHT API Insert(key, value, numReplicas); adds a

numReplicas argument to insert. Ensures resilience of the tuple to up to numReplicas-1 “simultaneous” node failures.

Open questions: consistency

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Caching DHT API

Same as basic DHT API. Implementation uses dynamic caching to balance query load.

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Resilient DHT API

Same as replicating DHT API. Implementation uses dynamic caching to balance query load.

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Publish API: Overview Publish(key, object); object = Lookup(key); Remove(key, object):

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Publish API Publish(key, object): ensures that the locally

stored object can be located using the key. Multiple instances of the object may be published under the same key from different locations.

object = Lookup(key): locates the nearest instance of the object associated with key. Returns null if no such object exists.

IPTPS 2003 ravenben@eecs.berkeley.edu February 20, 2003

Publish API Remove(key, object): after this operation

completes, the local instance of object can no longer be located using key.