Peer-to-Peer Systems - tsn.hhu.de · e.g. SOPCAST, PPLive and Zattoo. Two approaches for video streaming ! Tree-based overlay • Push content delivery • Single or multiple tree

Peer-to-Peer Systems

Winter semester 2014 Jun.-Prof. Dr.-Ing. Kalman Graffi Heinrich Heine University Düsseldorf

2 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html

Small Exercise: Bloom Filters 1

Consider a Bloom Filter consisting of §  an array of size m and §  k hash functions.

Initially all the bits of the array are set to zero. Assume that each position of the array can be selected by

each hash function with equal probability. Calculate the probability that a certain bit will be set to 1 after inserting n elements.


Small Exercise: Bloom Filters 1 - Solution

The probability that a certain bit is not set to one by a certain hash function during the insertion of an element is: 1 − 1/m

The probability that it is not set by any of the hash functions

is: (1 − 1/m)^k If we have inserted n elements, the probability that a certain

bit is still 0 is: (1 − 1/m)^kn the probability that it is 1 is therefore: 1 − (1 − 1/m)^kn


Small Exercise: Bloom Filters 2

Consider a Bloom Filter consisting of §  an array of size m = 5 §  K = 2 hash functions

H1(x) = x mod 5 H2(x) = (2 ∗ x + 3) mod 5. Initially the bit vector is unset (each element equal to 0). Insert 9 and 11 in the Bloom Filter. Show the state of each bit before and after each insertion.


Small Exercise: Bloom Filters 2 - Solution

Calculate the outcome of each hash function for 9 and 11. H1(9) = 4 H2(9) = 1 H1(11) = 1 H2(11) = 0 Check the membership for the values 15 and 16. Calculate the outcome of each hash function for 9 and 15. 15: H1(15) = 0 H2(15) = 3 Bit at position 3 is set to 0, 15 is not a member. (correct) 16: H1(16) = 1 H2(16) = 0 Both bits are set to 1. (false positive)

H1(x) = xmod5H2(x) = (2 ⇤ x +3)mod5

H1(9) = 4H2(9) = 1H1(11) = 1H2(11) = 0

H1(15) = 0H2(15) = 3

H1(16) = 1H2(16) = 0

Peer-to-Peer Systems – Chapter 9

P2P Streaming P2P Storage Fundamentals


Content Distribution and Multimedia Streaming – P2P Live and Video-on-Demand Streaming

This slide set is based on the lecture "Communication Networks 2" of Prof. Dr.-Ing. Ralf Steinmetz at TU Darmstadt


P2P Live and Video-on-Demand Streaming

Live P2P streaming has become §  increasingly popular approach for

streaming live content §  many receivers, similar to IPTV §  e.g. SOPCAST, PPLive and Zattoo.

Two approaches for video streaming

§  Tree-based overlay •  Push content delivery •  Single or multiple tree

§  Mesh-shaped overlay •  Pull content delivery (swarming) •  Like BitTorrent with modified chunk selection

Research issues: §  Streaming topology §  Scalable video codecs (more bandwidth, better quality)


BBC iPlayer §  P2P-based video streaming

platform

OctoShape §  P2P live streaming

application §  Peers receiving fragments of

live stream from other peers Results from ESC 2006

§  140.000 viewers from 140 countries

§  High quality stream (700kbps)

§  Claim: Traffic savings of 97% compared to C/S system

à Estimation: 53% – 79% traffic savings


Peer-to-Peer TV: Zattoo

Started at 2006 FIFA World Championship 2006

with 4 Swiss TV channels


Tree-based Video Streaming

Each peer receives the video stream from his parent Each peer forwards the stream to its children Initial content streamed from server Content is pushed through the network Pros:

§  High efficiency §  Low delay

Cons: §  Not very resilient to churn §  Upload requirements high


Impact of Peer Churn on Tree based Structure

Churn destroys tree structure §  Mechanisms for robustness needed §  Can be complex and tricky


Mesh-based Video Streaming

Idea Similar to BitTorrent, peers download the content from each other. Uses modified chunk selection

§  Higher priority for chunks that are about to be played.

Content is pulled from the network Pros

§  More resilient to churn §  Upload requirements flexible

Cons

§  Redundant chunks §  Lower efficiency than tree based topology §  High delay, due to exchange of buffer maps.


Data Pull in Mesh-based Video Streaming

Data Pull §  Buffer map is a map of all available chunks at a certain peer. §  Exchanged periodically to announce available chunks.


Block Selection

High priority set §  Sliding buffer window, updated §  with playback position

Download task dispatching

§  Parallel download from multiple peers §  Keep all peers as busy as possible

Priority calculation

§  High priority set: use greedy approach •  Chunks close to playback position and base layer get highest priority

§  Low priority zone: use non-greedy approach •  Download blocks “soon most wanted” by receiver set

Buffering Window

Playing Window

Low priority

t


Content Distribution and Multimedia Streaming – Adaptive P2P Video Streaming

This slide set is based on the lecture "Communication Networks 2" of Prof. Dr.-Ing. Ralf Steinmetz at TU Darmstadt


Motivation for Adaptive Video Streaming

Video is a major part of Internet traffic §  By 2014 almost 90% of

Internet traffic Peer-assisted solution

§  Reduce server load by making use of client-side resources

§  System is more scalable

But how to overcome weak peer contributions and heterogeneity? §  Scalable Video Coding

(SVC) can relieve and solve many issues and problems

Bandwidth Screen size CPU Home PC 10 Mbps 1920x1080 >2 GHz

Netbook 5 Mbps 1024 x 600 1.6 GHz

Smart phone 1 – 2 Mbps 800 x 480 900 MHz


Main Challenges for (Adaptive) P2P Streaming

Peer selection §  Select peers that can provide the required quality using the

available bandwidth §  From which peers to retrieve a chunk

Coordinate delivery from the peers §  Sender-Receiver coordination §  Which peer to give chunks?

Chunk selection §  Which chunks to request / to give?

Handle unpredictable variations in bandwidth §  Using fast layer switching §  What to do if chunks are missing at playback time?

Packet assignment §  Based on network and route status from each sender


Adaptive Video Streaming over P2P

Motivation §  Client/Server: scalable only for low resolution content (e.g. YouTube) §  P2P: works only when there are enough peers online

Solution and vision §  Hybrid solution using adaptive video streaming

Adaptive video streaming §  Multi-layer codec (e.g.: H264 Advanced Video Coding)

•  Base layer always needed (streamed from server) •  More layers à higher quality

–  Support heterogeneity –  Always on service and scalable

Underlay-aware overlay construction: §  Awareness for delay, locality, bandwidth §  How to gather the required information about the underlay

•  Requirements for Future Internet


SVC: Block-based Quality Scalability

Three-dimensional scalability

SVC Cube-Model §  Each GOP is modeled by a 3D-cube §  Block-combinations form layers §  Base layer is the most important

Must consider §  Interdependencies of blocks §  Deadline of blocks §  User preference

Base Tier (0)

Tier 2 Tier 2

Tier 3 Tier 3

Tier 3

Tier 1 Tier 2

Tier 1 Tier 1 Tier 2

Tier 3 Tier 3 Tier 3

Spa1

al Resolu1

on (D

)

Temporal Resolu1on (T)


Scalable Video Coding: Flavors of Scalability

Temporal Scalability: Different frame rate, e.g §  7.5 fps used in mobile phones. §  15 fps used in Netbooks, iPhone,... §  30 fps used in PCs

Spatial Scalability: Different resolution. §  QCIF, CIF, TV

Quality scalability §  Different quality levels.

Quality adaptation approaches Initial Adaptation

§  Support heterogeneous devices §  Different memory, processing and bandwidth profiles

Progressive Adaptation §  Support adaptation to changing underlay conditions §  Bandwidth, delay,...

Picture sources: http://www.hhi.fraunhofer.de/index.php?id=2767&L=1


Quality Adaptation Algorithms

Select SVC layer according to •  Peer resources and network dynamics •  Different strategy depending on the stage of the streaming session

Initial Quality Adaptation

Chunk/Block Selection

Progressive Quality Adaptation

Streaming

Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"


Initial Quality Adaptation (IQA)

Determines stream-able SVC layers §  Based on static peer resources §  Invoked at the beginning of

video playback Goal

§  Avoid long startup times §  Match resources at session start

Uses static peer resources

Stages §  Spatial adaptation §  Bit-rate adaptation §  Complexity adaptation



Progressive Quality Adaptation (PQA)

Adapt to real time changes of the network §  Activated periodically (every T

seconds) §  Based on real time network

information Goal

§  Predict possible stalls before they happen

§  Avoid stalls by temporary switching the layer

Uses real time information

Stages §  Net-status adaptation §  Bitrate adaptation §  Complexity adaptation



PQA Stages

Net-status Adaptation §  Only request layers that are available within local neighborhood

•  Determine the highest supported SVC layer •  Avoid waiting for rare blocks by temporary switching the layer

Bit-rate Adaptation §  Adjust layer according to throughput of high priority set §  Throughput observed through the fullness of the high priority set §  Avoid buffer under run by switching down the layer when

throughput is low §  Switch layer up in case throughput high enough

Complexity Adaptation §  Use models that estimate required processing speed for decoding

each layer



Peer Selection and Neighbor Management

Tracker manages: active peers and layers they currently stream §  Important in order to connect correct providers and consumers

Peers advertise currently streamed SVC layers §  After successful IQA/PQA to the tracker and neighbors §  During connection establishment phase with neighbors §  Buffer maps are extended to support SVC

The mechanism is bi-directional

§  Peers are eventually clustered according to their resources §  Seeders/caches support both weak and strong peers.



Evaluation Metrics

Session Quality §  Relative playback delay

Video Quality

§  Number of layer changes §  Relative received layer

§  Less layer changes or higher relative received layer à Better user experience



Impact of Quality Adaptation: Session Quality

0 10 20 30 40 50 60 70 80 900

20

40

60

80

100

120

140

160

180

200

without IQA, PQAwith only IQAwith IQA and PQA

Weak Peers Medium Peers Strong Peers

Tota

l pla

ybac

k de

lay

(Sec

onds

)



Impact of PQA Frequency: Visualization

§  The darker, the higher is the layer. White indicates a stall

PQA every 5 seconds PQA every 30 seconds

Stro

ng

M

ediu

m

W

eak



Small Exercise: Clustering Coefficient

Calculate the clustering coeffcient for the nodes: 3, 9 and 12. Calculate also the diameter of the network. CC = (Number of links between the vertices within its neighbourhood)/ (number of links that could possibly exist between them)

Ci vi

4 · 3/2 = 6 CC = 1/64 · 3/2 = 6 CC = 2/6 = 1/36 · 5/2 = 15 CC = 3/15 = 1/5


Small Exercise: Clustering Coefficient - Solution

Calculate the clustering coeffcient for the nodes: 3, 9 and 12. For number 3: 4 neighbors, 1 link, 4·3/2 = 6 possible links, CC = 1/6 For number 9: 4 neighbors, 2 links, 4·3/2 = 6 possible links, CC = 2/6 = 1/3 For number 12: 6 neighbors, 3 links, 6·5/2 = 15 possible links,CC = 3/15 = 1/5 Diameter: maximum of all shortest routes between all nodes. In this case the diameter is 5, e.g. 1-3-7-9-15-16 Ci vi

4 · 3/2 = 6 CC = 1/64 · 3/2 = 6 CC = 2/6 = 1/36 · 5/2 = 15 CC = 3/15 = 1/5


P2P-based Storage Systems – Overview on P2P Filesystem Approaches

This slide set is based on the lecture “Peer-to-Peer Networks" of Prof. Dr. Jussi Kangasharju at University of Helsinki


How to use a DHT?

Recall: DHT maps keys to values Applications based on DHTs must need this functionality

§  Or: Must be designed in this way! §  Possible to design an application in several ways

Keys and values are application specific

§  For filesystem: Value = file §  For email: Value = email message §  For distributed DB: Value = contents of entry, etc. §  For distributed data structures:

•  Value = end item / file •  Or value = structured set of further keys

Application stores values in DHT and uses them

§  Simple, but a powerful tool


Problems Solved by DHT

DHT solves §  The problem of mapping keys to values in the distributed hash

table

Implications

§  Efficient storage and retrieval of values / objects §  Efficient routing

•  Robust against many failures •  Efficient in terms of network usage

Provides hash table-like abstraction to application


Problems NOT Solved by DHT

Everything else except what is on previous slide… In particular, the following problems

§  Functionality •  DHT is: Write-Once-Read-Many •  No support for (multi-user) value modification, deletion

§  Robustness •  No guarantees against big failures •  Threat models for DHTs not well-understood yet

§  Availability •  Data not guaranteed to be available •  Only probabilistic guarantees (but possible to get high prob.)

§  Consistency •  No support for consistency (=same view on object for every) •  Data in DHT often highly replicated, consistency is a problem

§  Version management •  No support for version management •  Might be possible to support this to some degree


P2P Filesystems

P2P filesystems (FS) or P2P storage systems §  File storage and retrieval §  First applications of DHTs

Fundamental principle:

§  Key = filename, Value = file contents

Different kinds of systems

§  Storage for read-only objects §  Read-write support §  Stand-alone storage systems §  Systems with links to standard filesystems (e.g., NFS)


Design Space of P2P Storage

Slide based on Anwitaman Datta, “Peer�-to-Peer Storage Systems: Crowdsourcing the storage cloud“, ICDCN ‘10


P2P Filesystems: Basic Techniques

Fundamental basic techniques for building distributed storage systems §  Splitting files into blocks §  Replicating files (or blocks!) §  Using logs to allow modifications

For now: Simple analysis of advantages and disadvantages and three examples


Splitting Files into Blocks

Motivation §  Files are of different sizes and peers storing large files have to serve

more data Benefits

§  Load balancing can be achieved by •  Dividing files into equal-sized blocks •  Storing blocks on different peers

§  If different files share blocks, we can save on storage Drawbacks:

§  Instead of needing one peer online, all peers with all blocks must be online

§  Need metadata about blocks to be stored somewhere §  Granularity tradeoff:

•  Small blocks à Good load balance, lots of overhead •  Large blocks à Bad load balance, small overhead


Replication

Motivation §  If file (or block) is stored only on one peer and that peer is offline,

data is not available §  Replicating content to multiple peers significantly increases

content availability Benefits

§  High availability and reliability §  But only probabilistic guarantees

Con:

§  How to coordinate lots of replicas? •  Especially important if content can change (or can be deleted)

§  Good availability in unreliable networks requires high degree of replication

•  “Wastes” storage space


Change Logs

Motivation §  If we want to change the stored files, we need to modify every

stored replica §  Keep a log for every file (user, …)

•  Log gives information about the latest version

Benefits

§  Changes concentrated in one place §  Anyone can figure out what is the latest version

Drawbacks

§  How to keep the log available? §  By replicating it? ;-)


Unstructured Solutions

Motivation §  Distributed data structures are harmed by churn §  Idea: skip structure, have clever search mechanisms

Benefits §  Easy to setup and implement §  Good in small scenarios

•  Example: former p2p collaboration tool Groove (up to 10 users)

Drawbacks §  Large search overhead comes with full retrievability

•  Turns solution not scalable §  Too simple for many applications


P2P Filesystems: Open Issues

Capacity of hybrid architecture, and efficient schemes §  How much centralized provisioning is necessary for a given

distribution of peer resources?

Redundancy placement issues §  Cover 24 hours §  Minimize storage cost & access latency

Adaptive redundancy management §  Different data may need different QoS

Fine‐grained access control §  Example scenarios from P2P-based online social networks

Heterogeneity, energy efficiency, load‐balance etc.


P2P Filesystems: Real-World Examples

Microsoft has done lot of research in this area §  Even built prototypes

Why no products on the market?

§  Below some wild speculation §  P2P FS would compete with traditional file servers?

•  P2P needs to be built in to the OS, would kill server market? •  Who would build such an OS?

–  No big companies. But small companies are to small? Academia? –  Building “too much” software in academia: not research?

§  Impossible to build a “good enough” P2P FS? •  Theoretically doable, but too slow and weak in practice?

§  P2P filesystem can be done, but has no advantages? •  Solving one task (file system) opens many new requirements

–  Monitoring, quality control, security and access control, privacy, ability to delete criminal content, accounting, quota enforcement, …

–  Many tasks simple to solve centralized, but hard to solve decentralized

Documents

Peer-to-Peer Systems - tsn.hhu.de · e.g. SOPCAST, PPLive and Zattoo. Two approaches for video streaming ! Tree-based overlay • Push content delivery • Single or multiple tree