TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Topology Management for UnstructuredOverlay Networks

Joao Leitao

September 5th, 2012

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Roadmap

1 Introduction

2 Overview

3 CellFarm

4 X-BOT

5 Thicket

6 OpenFire

7 Conclusions

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionP2P Systems

Peer-to-Peer model:

Promise to overcome the limitations of the client-servermodel.

Fault-Tolerance: There is no single point of failure.

Inherent Scalability: All nodes contribute with theirresources.

Popular Examples:

File Sharing: Napster, Emule, Gnutella, Bittorrent.

VOIP: Skype.

IPTV: PPLive.

Internet Anonymity: TOR.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionP2P Systems

Peer-to-Peer model:

Promise to overcome the limitations of the client-servermodel.

Fault-Tolerance: There is no single point of failure.

Inherent Scalability: All nodes contribute with theirresources.

Popular Examples:

File Sharing: Napster, Emule, Gnutella, Bittorrent.

VOIP: Skype.

IPTV: PPLive.

Internet Anonymity: TOR.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionP2P Global Membeship

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionP2P Global Membeship

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionP2P Global Membership

Global Membership:

Limits scalability.High maintenance cost under high membership changes(e.g., churn).

Alternative is to rely on a (distributed) membershipservice:

Offers a partial view to each participant.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionP2P Global Membership

Global Membership:

Limits scalability.High maintenance cost under high membership changes(e.g., churn).

Alternative is to rely on a (distributed) membershipservice:

Offers a partial view to each participant.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionPartial Views

Partial Views:

Encodes neighboring relations across participants.

Their closure establishes an overlay network.

Topology properties can affect P2P services performance:

Efficiency.

Fault-tolerance.

Convergence.

Latency.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionPartial Views

Partial Views:

Encodes neighboring relations across participants.

Their closure establishes an overlay network.

Topology properties can affect P2P services performance:

Efficiency.

Fault-tolerance.

Convergence.

Latency.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionOverlay Networks

Two types of overlay networks:

Structured Overlay Networks (e.g., DHTs)

Unstructured Overlay Networks (i.e., random overlays).

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionStructured Overlay Networks

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionStructured Overlay Networks

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionUnstructured Overlay Networks

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionOverlay Networks

Structured Overlay Networks

Can (easily) offer additional functionality to services.

Complex construction and maintenance.

More susceptible to (high) membership dynamics.

Unstructured Overlay Networks

Lower construction and maintenance costs.

More robust to (high) membership dynamics.

Natural redundancy.

Topology cannot be easily leveraged by services.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionOverlay Networks

Structured Overlay Networks

Can (easily) offer additional functionality to services.

Complex construction and maintenance.

More susceptible to (high) membership dynamics.

Unstructured Overlay Networks

Lower construction and maintenance costs.

More robust to (high) membership dynamics.

Natural redundancy.

Topology cannot be easily leveraged by services.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionOverlay Networks

Structured Overlay Networks

Can (easily) offer additional functionality to services.

Complex construction and maintenance.

More susceptible to (high) membership dynamics.

Unstructured Overlay Networks

Lower construction and maintenance costs.

More robust to (high) membership dynamics.

Natural redundancy.

Topology cannot be easily leveraged by services.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionOverlay Networks

Structured Overlay Networks

Can (easily) offer additional functionality to services.

Complex construction and maintenance.

More susceptible to (high) membership dynamics.

Unstructured Overlay Networks

Lower construction and maintenance costs.

More robust to (high) membership dynamics.

Natural redundancy.

Topology cannot be easily leveraged by services.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionProblem Statement

Motivation:

Get the best of both worlds by imbuing some form of relaxedstructure over unstructured overlay networks.

Retain the simplicity, flexibility, and robustness ofunstructured overlays.

Offer some topology properties than can benefit and beleveraged by P2P services and applications.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionProblem Statement

Motivation:

Get the best of both worlds by imbuing some form of relaxedstructure over unstructured overlay networks.

Retain the simplicity, flexibility, and robustness ofunstructured overlays.

Offer some topology properties than can benefit and beleveraged by P2P services and applications.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

IntroductionProblem Statement

Motivation:

Get the best of both worlds by imbuing some form of relaxedstructure over unstructured overlay networks.

Retain the simplicity, flexibility, and robustness ofunstructured overlays.

Offer some topology properties than can benefit and beleveraged by P2P services and applications.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Roadmap

1 Introduction

2 Overview

3 CellFarm

4 X-BOT

5 Thicket

6 OpenFire

7 Conclusions

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewP2P Typical Architecture

Applica'on*

Peer.to.Peer*Service*

Overlay*Network*

Network*

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewControl Technique

Applica'on*

Peer.to.Peer*Service*

Overlay*Network****

Network*

Control%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewControl Technique

Soft-constraints over neighboring relations.

Enforce topological properties.During the construction of the overlay.

Maintain high level of randomness:

Overlay is flexible to deal with churn.Keep maintenance overhead low.

Example:

Generate dynamic cliques of nodes that can be exploited tosupport replication and load balancing.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewControl Technique

Soft-constraints over neighboring relations.

Enforce topological properties.During the construction of the overlay.

Maintain high level of randomness:

Overlay is flexible to deal with churn.Keep maintenance overhead low.

Example:

Generate dynamic cliques of nodes that can be exploited tosupport replication and load balancing.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewControl Technique

Soft-constraints over neighboring relations.

Enforce topological properties.During the construction of the overlay.

Maintain high level of randomness:

Overlay is flexible to deal with churn.Keep maintenance overhead low.

Example:

Generate dynamic cliques of nodes that can be exploited tosupport replication and load balancing.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewBias Technique

Applica'on*

Peer.to.Peer*Service*

Overlay*Network****

Network*

Bias(Control%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewBias Technique

Build an random overlay network.

Iteratively:

Swap some existing overlay links by other links that arebetter.Consider a particular performance criteria (e.g., latency).

Resulting overlay:

Random in nature.Optimized to benefit the operation of a P2P service.

Example:

Promote low-latency overlay links to improve the latency of anapplication-level broadcast service.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewBias Technique

Build an random overlay network.

Iteratively:

Swap some existing overlay links by other links that arebetter.Consider a particular performance criteria (e.g., latency).

Resulting overlay:

Random in nature.Optimized to benefit the operation of a P2P service.

Example:

Promote low-latency overlay links to improve the latency of anapplication-level broadcast service.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewBias Technique

Build an random overlay network.

Iteratively:

Swap some existing overlay links by other links that arebetter.Consider a particular performance criteria (e.g., latency).

Resulting overlay:

Random in nature.Optimized to benefit the operation of a P2P service.

Example:

Promote low-latency overlay links to improve the latency of anapplication-level broadcast service.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewEmbed Technique

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Embed%

Bias(Control%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewEmbed Technique

Considering feedback from a P2P service.

Embed a secondary topology over a pure unstructuredoverlay network:

Secondary topology can be leveraged by the service.

Remaining overlay links can be used to transmit controlinformation:

Recover secondary topology in face of failures.Fallback if secondary topology becomes compromised.

Example

Embed spanning trees over an unstructured overlay to supportmultimedia streaming.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewEmbed Technique

Considering feedback from a P2P service.

Embed a secondary topology over a pure unstructuredoverlay network:

Secondary topology can be leveraged by the service.

Remaining overlay links can be used to transmit controlinformation:

Recover secondary topology in face of failures.Fallback if secondary topology becomes compromised.

Example

Embed spanning trees over an unstructured overlay to supportmultimedia streaming.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewEmbed Technique

Considering feedback from a P2P service.

Embed a secondary topology over a pure unstructuredoverlay network:

Secondary topology can be leveraged by the service.

Remaining overlay links can be used to transmit controlinformation:

Recover secondary topology in face of failures.Fallback if secondary topology becomes compromised.

Example

Embed spanning trees over an unstructured overlay to supportmultimedia streaming.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewEnrich Technique

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich%Embed%

Bias(Control%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewEnrich Technique

Considering feedback from a P2P service.

Create temporary overlay links at the P2P service layer:

Temporary links are outside the logic of the unstructuredoverlay.Enable the service to contact peers that could not beavailable through the overlay.

Example

Create temporary overlay links that enable one to circumventFirewalls and NAT boxes.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Overview

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich%Embed%

Bias(Control%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewCellFarm

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich%Embed%

Bias(Control:(CellFarm%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewX-BOT

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich%Embed%

Bias:(X/BOT%

Control:(CellFarm%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewThicket

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich(Embed:(Thicket%

Bias:(X/BOT%

Control:(CellFarm%

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OverviewOpenFire

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Overview

Very briefly motivate the problem and solution.

Present some relevant results.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Overview

Very briefly motivate the problem and solution.

Present some relevant results.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Control the Topology

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarm

Goal: Offer support to replication and load balancing:

Process groups.

This can be achieved by manipulating the topology:

Nodes self-organize in (fully connected) cliques.Cliques are used as a replication and load balancing unit.Cliques are highly connected among them.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarm

Goal: Offer support to replication and load balancing:

Process groups.

This can be achieved by manipulating the topology:

Nodes self-organize in (fully connected) cliques.Cliques are used as a replication and load balancing unit.Cliques are highly connected among them.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarm

Employed technique: Control.

Flexibility is essential:

Enable the overlay to cope with churn.Clique size governed by three parameters: target size;minimum size; maximum size;

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarm

Employed technique: Control.

Flexibility is essential:

Enable the overlay to cope with churn.Clique size governed by three parameters: target size;minimum size; maximum size;

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarmMain Results: Cell Size Distribution

0"

1"

2"

3"

4"

5"

6"

8" 9" 10" 11" 12" 13" 14" 15" 16"

Num

ber'o

f'Cells'

Cell'Size''

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarmMain Results: Robustness to Churn

0"

10"

20"

30"

40"

50"

60"

70"

80"

90"

100"

0.1" 0.2" 0.3" 0.4" 0.5" 1" 2" 3" 4" 5"

Num

ber"o

f"nod

es"in"largest"con

nected

"compo

enen

t"(%)"

Churn"rate"(%)"

CellFarm(

Unstrctured(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarmSummary

CellFarm, based on the control technique operating at theoverlay network layer.

Control technique allows to build and maintain robust andflexible overlays with pre-determined topologicalproperties.

Benefit the operation of P2P resource location systemsbased on unstructured overlays and one-hop replication.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

CellFarmSummary

CellFarm, based on the control technique operating at theoverlay network layer.

Control technique allows to build and maintain robust andflexible overlays with pre-determined topologicalproperties.

Benefit the operation of P2P resource location systemsbased on unstructured overlays and one-hop replication.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Overview

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Bias the Topology

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOT

Goal: Deal with the topology mismatch problem:

Random topology means that many links may besub-optimal.

This can be achieved by manipulating the topology:

Cannot employ control technique.New nodes may offer new opportunities to improve theoverlay topology.Allow nodes to replace sub-optimal links by better links.Approach can be used to improve the overlay for differentcriteria.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOT

Goal: Deal with the topology mismatch problem:

Random topology means that many links may besub-optimal.

This can be achieved by manipulating the topology:

Cannot employ control technique.New nodes may offer new opportunities to improve theoverlay topology.Allow nodes to replace sub-optimal links by better links.Approach can be used to improve the overlay for differentcriteria.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOT

Employed technique: Bias.

Generic approach: Companion oracle tags overlay linkswith costs.

X-BOT strives to minimize the overlay cost.

Some minimal coordination is essential:

Protect relevant properties of an unstructured overlay(e.g., connectivity).Rely on 4-node coordination technique to bias the overlay.X-BOT also includes other mechanisms to protect relevantoverlay properties.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOT

Employed technique: Bias.

Generic approach: Companion oracle tags overlay linkswith costs.

X-BOT strives to minimize the overlay cost.

Some minimal coordination is essential:

Protect relevant properties of an unstructured overlay(e.g., connectivity).Rely on 4-node coordination technique to bias the overlay.X-BOT also includes other mechanisms to protect relevantoverlay properties.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOT

Employed technique: Bias.

Generic approach: Companion oracle tags overlay linkswith costs.

X-BOT strives to minimize the overlay cost.

Some minimal coordination is essential:

Protect relevant properties of an unstructured overlay(e.g., connectivity).Rely on 4-node coordination technique to bias the overlay.X-BOT also includes other mechanisms to protect relevantoverlay properties.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOTMain Results: Overlay Cost

0 50 100 150 200 250cycle

0

5x107

1x108

1.5x108

2x108ov

erla

y co

stT-ManAraneolaGoCastX-BOT

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOTMain Results: Broadcast Latency & Reliability

Gossip-based BroadcastLatency (ms) Reliability (%)

Araneola 3517.0 100.00000GoCast 2108.00 99, 99996T-Man 2545.2 13.80600X-BOT 1879.8 100.00000

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOTMain Results: Broadcast Latency

Inet-3.0 ScenarioLatency (ms) Reliability (%)

Araneola 3517.0 100.00000GoCast 2108.00 99, 99996T-Man 2545.2 13.80600X-BOT 1879.8 100.00000

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOTSummary

X-BOT, based on the bias technique operating at theoverlay network layer.

Bias technique allows to achieve improved overlaytopologies that can better match the requirements ofservices executing on top of them.

If a minimal amount of coordination exists among nodes,biasing can be achieved without compromising relevantoverlay properties.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

X-BOTSummary

X-BOT, based on the bias technique operating at theoverlay network layer.

Bias technique allows to achieve improved overlaytopologies that can better match the requirements ofservices executing on top of them.

If a minimal amount of coordination exists among nodes,biasing can be achieved without compromising relevantoverlay properties.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Bias the Topology

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Embed the Topology

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Thicket

Goal: Support efficient and robust data streaming:

Efficient: Rely on spanning tree.Robust: Multiple spanning trees.Load distribution: Multiple interior-node disjoint trees.

Efficient spanning tree topologies should consider feedbackof streaming service:

Cannot employ a technique at the overlay layer.Building independent trees is not an option.Use a single unstructured overlay networks as a base.Embed several spanning trees over that overlay in acoordinated fashion.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Thicket

Goal: Support efficient and robust data streaming:

Efficient: Rely on spanning tree.Robust: Multiple spanning trees.Load distribution: Multiple interior-node disjoint trees.

Efficient spanning tree topologies should consider feedbackof streaming service:

Cannot employ a technique at the overlay layer.Building independent trees is not an option.Use a single unstructured overlay networks as a base.Embed several spanning trees over that overlay in acoordinated fashion.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Thicket

Employed technique: Embed.

Coordination is essential.

Overlay links used to embed a tree should consider theexistence of other trees.Trees should be able to recover from partitions efficiently.The maximum forwarding load imposed over nodes shouldbe limited.

Multiples trees allow to transmit redundant data:

e.g., Network coding techniques.Sustain the data stream even if one tree becomestemporarily partitioned.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Thicket

Employed technique: Embed.

Coordination is essential.

Overlay links used to embed a tree should consider theexistence of other trees.Trees should be able to recover from partitions efficiently.The maximum forwarding load imposed over nodes shouldbe limited.

Multiples trees allow to transmit redundant data:

e.g., Network coding techniques.Sustain the data stream even if one tree becomestemporarily partitioned.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Thicket

Employed technique: Embed.

Coordination is essential.

Overlay links used to embed a tree should consider theexistence of other trees.Trees should be able to recover from partitions efficiently.The maximum forwarding load imposed over nodes shouldbe limited.

Multiples trees allow to transmit redundant data:

e.g., Network coding techniques.Sustain the data stream even if one tree becomestemporarily partitioned.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

ThicketMain Results: K-interior node distribution (PlanetLab)

0

20

40

60

80

100

0 1 2 3 4 5

num

ber o

f int

erio

r nod

es (%

)

number of trees

Thicket

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

ThicketMain Results: Streaming service robustness in face of targeted node failures

60

65

70

75

80

85

90

95

100

105

0 50 100 150 200

relia

bilit

y (%

)

simulation cycle

NUTSBOLTSThicket

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

ThicketSummary

Thicket, based on the embed technique operating at thepeer-to-peer service layer.

Embed technique allows to efficiently embed interior-nodedisjoint trees over a single unstructured overlay.

Combines the best of gossip-based and tree-baseddissemination approaches.

Improved the usage of available resources when comparedwith existing solutions.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

ThicketSummary

Thicket, based on the embed technique operating at thepeer-to-peer service layer.

Embed technique allows to efficiently embed interior-nodedisjoint trees over a single unstructured overlay.

Combines the best of gossip-based and tree-baseddissemination approaches.

Improved the usage of available resources when comparedwith existing solutions.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Embed the Topology

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Enrich the Topology

Applica'on*

Peer.to.Peer*Service****

Overlay*Network****

Network*

Enrich:(OpenFire(

Embed:(Thicket(

Bias:(X/BOT(

Control:(CellFarm(

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFire

Goal: Deal with the presence of Firewalls and NAT boxes.

Firewalls and NAT boxes make nodes unreachable.This can produce an unbalanced behavior in P2P services(e.g, rumor mongering protocols).Most nodes interact only with public nodes in the Internet.Those nodes can easily became resource exhausted.

Circumvent Firewalls and NAT boxes:

Taking into consideration communication patters at theP2P service layer.Keep connections established by confined nodes open to beused them later.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFire

Goal: Deal with the presence of Firewalls and NAT boxes.

Firewalls and NAT boxes make nodes unreachable.This can produce an unbalanced behavior in P2P services(e.g, rumor mongering protocols).Most nodes interact only with public nodes in the Internet.Those nodes can easily became resource exhausted.

Circumvent Firewalls and NAT boxes:

Taking into consideration communication patters at theP2P service layer.Keep connections established by confined nodes open to beused them later.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFire

Employed technique: Enrich.

Use a single temporary overlay link.

Whenever a node receives a request from a peer it updatesits temporary overlay link.Nodes only process a (rumor mongering) request for eachrequest they sent themselves.Additional received requests are forwarded across thetemporary connection.

This solution is interesting:

Can easily ensure balanced processing of messages.Nodes are not required to be aware of Firewalls and NATsexplicitly.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFire

Employed technique: Enrich.

Use a single temporary overlay link.

Whenever a node receives a request from a peer it updatesits temporary overlay link.Nodes only process a (rumor mongering) request for eachrequest they sent themselves.Additional received requests are forwarded across thetemporary connection.

This solution is interesting:

Can easily ensure balanced processing of messages.Nodes are not required to be aware of Firewalls and NATsexplicitly.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFire

Employed technique: Enrich.

Use a single temporary overlay link.

Whenever a node receives a request from a peer it updatesits temporary overlay link.Nodes only process a (rumor mongering) request for eachrequest they sent themselves.Additional received requests are forwarded across thetemporary connection.

This solution is interesting:

Can easily ensure balanced processing of messages.Nodes are not required to be aware of Firewalls and NATsexplicitly.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFireMain Results: Maximum gossip exchanges per node

!" !#$$" %!$$" &#$$" #!$$" '#$$" (!$$" !$#$$" !)!$$"

!"#$%&'()'*(++,-'%./01!*%+'

!"#$%&'()'/(!2!%#%!3'4(#1,!+'

**+","!"

**+",")"

**+","-"

**+","!$"

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFireMain Results: Maximum latency

!"

#!"

$!"

%!"

&!"

'!!"

'#!"

'$!"

'%!"

'&!"

'" '%!!" ('!!" $%!!" %'!!" )%!!" *'!!" '!%!!" '#'!!"

!"#$%&'()*+$(,%-#./(

%,+0$1(23(&2%4%$+$%#(52+"-%.(

++,"-"'"

++,"-"#"

++,"-"."

++,"-"'!"

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFireSummary

OpenFire, based on the enrich technique operating at thepeer-to-peer service layer.

Enrich technique allows to overcome some of thechallenges introduced by Firewalls and NAT boxes.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

OpenFireSummary

OpenFire, based on the enrich technique operating at thepeer-to-peer service layer.

Enrich technique allows to overcome some of thechallenges introduced by Firewalls and NAT boxes.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Roadmap

1 Introduction

2 Overview

3 CellFarm

4 X-BOT

5 Thicket

6 OpenFire

7 Conclusions

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsConclusions

4 techniques to manage the topology of unstructuredoverlay networks.

Imbuing some relaxed form of structure.

Protecting the relevant properties of unstructuredoverlays:

Low overhead.Robustness to churn scenarios.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsConclusions

4 techniques to manage the topology of unstructuredoverlay networks.

Imbuing some relaxed form of structure.

Protecting the relevant properties of unstructuredoverlays:

Low overhead.Robustness to churn scenarios.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsConclusions

At the overlay network layer:

Control (CellFarm).Bias (X-BOT).

At the peer-to-peer service layer:

Embed (Thicket).Enrich (OpenFire).

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsConclusions

In the thesis:

Discusses the design of each solution in detail.Validates all proposed solutions.Experimentally evaluate their impact over particular casestudies.

Summary: The thesis has show and studied how to addsome form structure to unstructured overlay benefiting theoperation of peer-to-peer services.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsConclusions

In the thesis:

Discusses the design of each solution in detail.Validates all proposed solutions.Experimentally evaluate their impact over particular casestudies.

Summary: The thesis has show and studied how to addsome form structure to unstructured overlay benefiting theoperation of peer-to-peer services.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsFuture Research Directions

Improving volunteer parallel computing platforms

New decentralized volunteer computing infrastructures:

For instance based in solutions similar to CellFarm andX-BOT.Offer the opportunity to support more complex parallelprograming models (e.g, MapReduce).

Devise a new class of robust and flexible DHT overlaynetworks based on the X-BOT design.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsFuture Research Directions

Improving volunteer parallel computing platforms

New decentralized volunteer computing infrastructures:

For instance based in solutions similar to CellFarm andX-BOT.Offer the opportunity to support more complex parallelprograming models (e.g, MapReduce).

Devise a new class of robust and flexible DHT overlaynetworks based on the X-BOT design.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsFuture Research Directions

User-centric platform for social applications

Offer a decentralized platform to support a persistent anddynamic social network.

Providing the mechanisms for designing new collaborativeapplications over the social network that can operate in amore decentralized fashion.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Conclusions & Future Research DirectionsFuture Research Directions

User-centric platform for social applications

Offer a decentralized platform to support a persistent anddynamic social network.

Providing the mechanisms for designing new collaborativeapplications over the social network that can operate in amore decentralized fashion.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Publications

On the Structure of Unstructured Overlay Networks (fastabstract). J. Leitao, J. Pereira and L. Rodrigues. In Supplement ofthe 38th Annual IEEE/IFIP International Conference on DependableSystems and Networks, Anchorage, Alaska, USA, June, 2008.

Overnesia: a Robust Overlay Network for Virtual Super-Peers.Joao Leitao and Luıs Rodrigues. Technical Report 36/2009,INESC-ID, July 2009 (Available in:http: // www. inesc-id. pt/ ficheiros/ publicacoes/ 5510. pdf ).

X-BOT: A Protocol for Resilient Optimization of UnstructuredOverlays. J. Leitao, J. P. Marques, J. Pereira and L. Rodrigues.Proceedings of the 28th IEEE International Symposium on ReliableDistributed Systems, Niagara Falls, New York, U.S.A., Sep, 2009. pp.236–245.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Publications

On Adding Structure to Unstructured Overlay Networks. J. Leitao,N. Carvalho, J. Pereira, R. Oliveira, and L. Rodrigues. In Handbookof Peer-to-Peer Networking, X. Shen, H. Yu, J. Buford, M. Akon(Eds.), Springer 2010. pp. 327-365. ISBN: 978-0-387-09750-3.

Balancing Gossip Exchanges in Networks with Firewalls. J. Leitao,R. van Renesse and L. Rodrigues. Proceedings of the 9thInternational Workshop on Peer-to-Peer Systems (IPTPS ’10), SanJose, CA, USA, 27 April, 2010.

Thicket: A Protocol for Building and Maintaining Multiple Trees ina P2P Overlay. M. Ferreira, J. Leitao, and L. Rodrigues.Proceedings of the 29th IEEE Symposium on Reliable DistributedSystems (SRDS), New Delhi, India, 31 October-3 November 2010.

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Publications

X-BOT: A Protocol for Resilient Optimization of UnstructuredOverlay Networks. J. Leitao, J. P. Marques, J. Pereira, and L.Rodrigues. IEEE Transactions on Parallel and Distributed Systems(Published online January 2012).

Gossip-based Interior-Node-Disjoint Trees. J. Leitao, M. Ferreira, J.Pereira, and L. Rodrigues. IEEE Transactions on Parallel andDistributed Systems (Submitted).

TopologyManagement

forUnstructured

OverlayNetworks

Joao Leitao

Introduction

Overview

CellFarm

X-BOT

Thicket

OpenFire

Conclusions

Publications

Thanks for your attention.