Autonomous Replication for High Availability in Unstructured P2P Systems

Francisco Matias Cuenca-Acuna,

Richard P. Martin,

Thu D. Nguyen

Department of Computer Science,

Rutgers University

April, 2003

Content

Introduction Background: PlanetP Autonomous Replication Performance Evaluation Conclusions

Introduction (1)

Peer-to-peer (p2p) computing becomes a powerful paradigm sharing information across the internet

Problem of providing high availability for shared data Recent measurements suggest

– Members of p2p communities may be offline more than they are online

Providing practical availability, say 99-99.9% would be– Expensive storage-wise using traditional replication methods– Expensive bandwidth-wise as peers leave and rejoin the

community

Introduction (2)

Question: Is it possible to place replicas of shared files in such a way that

– Files are highly available– Without requiring the continual movement of replicas to

members currently online

Propose a distributed replication algorithm– Decisions are made entirely autonomously by individual

members– Only need a small amount of loosely synchronized global

state

Introduction (3)

Assumption– Files are replicated in their entirety only when a member

hoards that file for disconnected operation– Otherwise, files are replicated using an erasure code– Study a very weakly structured system because tight

coordination is difficult and costly in dynamic environment

Background: PlanetP (1)

PlanetP is a publish/subscribe system– Support content-based search, rank and retrieval

Members publish documents when they wish to share

Publication of a document– Give PlanetP a XML snippet containing a pointer to the file– PlanetP indexes the XML snippet and the file– Local index is used to support content search

Background: PlanetP (2)

Two major components to enable community-wide sharing

– An gossiping layer Periodically gossip about changes to keep shared data weakly

consistent

– Content search, rank and retrieval service Two data structures to be replicated on every peer

– Membership directory: Contains names and addresses of all current members

– Global content index: Contains term-to-peer mapping

Background: PlanetP (3)

To locate content– Users pose queries at a specific node– Identify the subset of target peers from the local copy of

global content index– Query is passed to these target peers– The targets evaluate the query against their local indexes

and return results (URLs for relevant documents)

Results have shown that PlanetP can easily scale to sizes of several thousands

Autonomous Replication (1)

Member’s hoard set– Members hoard some subset of the shared files entirely on

their local storage– Members take responsibility for ensuring the availability

Replicator– Member is trying to replicate an erasure-coded fragment of

a file Target

– Peer that the replicator is asking to store the fragment Replicator store

– Excess storage space contributed by each member for replication

Autonomous Replication (2)

Each file is identified by a unique ID Overall algorithm

– Advertises the file IDs in its hoard set and the fragments in its replication store to the global index

– Periodically estimates the availability of its hoarded files and the fragments (Estimating Availability)

– Every Tr time units, increase the availability of a file that is not at a target availability (Randomized Replication)

– The target peer saves the incoming fragment (Replacement Scheme)

Estimating Files Availability (1)

Replicated in two manners– Entire copies– Erasure-coded file fragments

H(f): set of peers hoarding a file f F(f): set of peers containing a fragment f A(f): availability of f

– All nodes in H(f) are simultaneously offline– At least n-m+1 of the nodes in F(f) are offline

)( fFn

Estimating Files Availability (2)

H(f) and F(f) do not intersect– Peer adds a file for which it is storing a fragment to its hoard

set, it ejects the fragment immediately A(f) does not account for the possibility of duplicate

fragments– n >> m

Randomized Replication (1)

Erasure codes (Reed Solomon) provide data redundancy– Divide a file into m fragment and recode them into n fragment (m <

n) Generate all n fragments => detect and regenerate specific lost

fragments Disadvantages for highly dynamic environment

– Member availability changes over time Necessary to change n => re-fragmenting and replication of some files

– Peers leaving Accurate accounting of which peer is storing which fragment => regenerate

fragment loss

– Peers temporarily going offline Introducing duplicate fragments

Randomized Replication (2)

Choose n >> m but do not generate all n fragments To increase the availability of a file

– RANDOMLY generate an additional fragment from the set of n possible fragments

Chance of having duplicate fragments is small if n is very large

Not having any peer coordination

Replacement (1)

A target peer receives a replication request If its store is full

– Decide whether to accept the incoming fragment, OR– select other fragments to evict from its store

Choose the fragments with the highest availability to make space

– Deterministic algorithm => victimize fragments of the same file => drastic changes in the file’s availability

Propose Weighted Random Selection Process

Replacement (2)

Policy– Compute the average number of nines in the availability of

the fragments– Incoming fragment’s number of nines > 10% of this average

=> reject incoming fragment– Lottery scheduling to select victim fragments

Divide tickets into two subsets with 80:20 Each fragment is assigned an equal share of the smaller subset Fragments with availability above 10% of the average are given a

portion of the larger subset

Replacement (3)

Example– Target node has 3 fragments with availability 0.99, 0.9, 0.5– Number of nines => 2, 1, 0.3– Average availability in nines + 10% = 0.76– If we have 100 tickets

First fragment=67+6.6, Second fragment=13+6.6, Third fragment=0+6.6

– Chances of each fragment to be evicted First fragment=0.74, Second fragment=0.2, Third fragment=0.06

Use number of nines rather than the availability– It linearizes the differences between values

Replicators (1)

Select fragment to replicate Similar to replacement, lottery scheduling is used

– Favoring files with low availability

Find target peer– Select a target randomly– If the target does not have sufficient space, select another

target– Repeat this process for five times– If not success, randomly choose from these five targets

Experimental Environment (1)

Event driven simulator Assumption

– Members replicate files at synchronous intervals– Not simulate the detail timing of message transfers– Not simulate the full PlanetP gossiping protocol

To account for the data staleness, reestimate file availability only once every 10 minutes

– Use Reed Solomon code and m=10

Experimental Environment (2)

Evaluate 3 different p2p communities File Sharing (FS) : very loosely coupled community

sharing multimedia files Commercial (CO): corporate/university environment Worgroup (WG): distributed development group

Experimental Environment (3)

Parameters– Per peer mean uptime and downtime– Peer arrival to and exit from the community as exponential

arrival process– Number of file per node– Number of hoarded replicas per file– Amount of excess space on each node– File sizes

BASE: nodes pushing and accepting replicas in the complete absences of information on file availability

OMNI: replication is driven by a central agent to maximize the minimum file availability

Availability VS Excess Storage

CO: small amount of excess space is required FS: 8X excess capacity for 3 nines availability File hoarding has little effect on file availability

(0 replication from hoarding) (25% of the files have 2 hoarded replicas) (25% of the files have 5 hoarded replicas)

Overall Availability (CDF)

(CO with 1X, 1.5X and 2X) (FS with 1X, 3X and 6X) (WG with 9X)

CO: 2X excess storage, over 99% of files with 3 nines availability FS: around 6X excess storage for 3 nines availability WG: performs better if files and excess storage are uniformly distributed

– Non-uniform => not easy for replicators to find free space on the subset of peers– Peers with the most files to replicate have the most excess storage

Against BASE

(CDF: Availability for FS with 3X) (BASE: number of fragments) (REP: number of fragments)

BASE: about 16% of files < a single nine availability Replacement policy can increase fairness BASE’s FIFO favors peers who are frequently online

– They push their files to less available peers even if the latter’s content should be replicated more

Bandwidth Usage

CO– REP: excess space from 1X to 2X => average number of

files replicated per hour from 10 to 0

FS– REP: excess space from 1X to 3X => average number of

files replicated per hour from 241 to 0– BASE: 3X excess space => replicate 816 files per hour

Conclusions

Address the question of increasing the availability of shared files

Study a decentralized algorithm under a very loosely coordinated environment

Achieve practical availability (99.9%) in a completely decentralized system with low individual availability

Such availability levels do not require– Sophisticated data structures– Complex replication schemes– Excessive bandwidth