Embed Size (px)
DESCRIPTION
Matthias Feldotto and Kalman Graffi. Comparative Evaluation of Peer-to-Peer Systems Using PeerfactSim.KOM. In IEEE HPCS’13: Proceedings of the International Conference on High Per- formance Computing and Simulation, 2013.
Citation preview
Comparative Evaluation of Peer-to-Peer Systems Using PeerfactSim.KOM
Kalman GraffiTechnology of Social Networks
Institute of Computer Science
Heinrich Heine University in Düsseldorf
eMail: [email protected]
Matthias FeldottoTheory of Distributed Systems
Institute of Computer Science
University of Paderborn
eMail: [email protected]
See: www.peerfact.org
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 2
Agenda
Background on Evaluation Approaches
Requirements on Simulating P2P Systems
PeerfactSim.KOM
Comparative Evaluation in PeerfactSim.KOM
Conclusions
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 3
Analysis– Often simplified model
• Homogeneous nodes, probabilistic actions– Leads to proofs (under specific conditions)
Example:– Given: Weighted DAG– Results:
• Proofs• Complexities - O (log N)
Good: General results
Weakness: – Details lost in abstraction– Sometimes the constants are important
General Evaluation Methods
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 4
Simulation– Advanced and heterogeneous model
• Specific node characteristics, capacities, behavior– Investigates emerging behavior– Often focus on quality of service
• Response times, induced traffic, specific node load• A response time of 1s to 5s matters!
Examples:
Given: – 10.000 nodes, capacity distribution X– 70% altruistic nodes, 20% selfish nodes, 10% malicious nodes– Protocol XY, workload Z
Results:– Statistics on quality of service over time
General Evaluation Methods
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 5
Prototype – in Testbed / in real world– Deployment of code in real testbed (e.g. PlanetLab)– Most adequate models, unpredictable user behaviour– Challenging to coordinate the tests, gather results– Logging and coordination might disturb the results
Example
Given: – 733 PlanetLab nodes– Full protocol stack: IP, TCP/UDP, middleware, application, virtual
users– Deployment in global PlanetLab
Results:– Behavior under realistic network conditions
• Delays, jitter, node load …
General Evaluation Methods
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 6
Agenda
Background on Evaluation Approaches
PeerfactSim.KOM
Requirements on Simulating P2P Systems
Comparative Evaluation in PeerfactSim.KOM
Conclusions
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 7
PeerfactSim.KOM (see www.peerfact.org)
Type– Event-based simulator– Written in Java– Simulations up to 100K peers possible– Focus on simulation of p2p systems on various layers
• User
• Application
• Services: monitoring, replication …
• Overlays
• Network models
Invitation to join the community– Several universities use and extend the simulator actively– Used and heavily extended in the project
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 8
Layered View
Layered Architecture– Easy exchange of components– Testing of new applications– Testing of new mechanisms
Main idea– Layers have several implementations– Enables testing of individual layer
mechanisms• on its own
• in combination with other layers
See www.peerfact.org
Application
Overlay
User
Sim
ulation E
ngin
e
Network
Service
Transport
Application
Overlay
User
Sim
ulation E
ngin
e
Network
Service
Transport
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 9
Layers and Implementations
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 10
Analyzers and Final Graph Generation
CSV filesCSV files
GraphsGraphs
gnuplot
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 11
Agenda
Background on Evaluation Approaches
PeerfactSim.KOM
Requirements on Simulating P2P Systems
Comparative Evaluation in PeerfactSim.KOM
Conclusions
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 12
Requirements on Simulating P2P Systems
For what do we want to use simulators?– Evaluation one or several protocol through– Parameter studies with multiple seeds
What we want: specify once – collect all graphs at the end– Specify all setups in the begin
• E.g. overlays=pastry,chord,kademlia network=simple,gnp seed=10
• Resulting in 60 simulations: 10 simulations each of pastry-simple, pastry-gnp, chord-simple, chord-gnp, kademlia-simple, kademlia-gnp
– Automated execution• Parallelity configurable, individual maximum memory
– Automated results processing and plotting• Combine all results of a setup – calculate means and std.deviations
• Plot results of all setups in graphs
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 13
Motivation and Essence of the Paper in Summary
Before
Simulationsetup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot resultsSimulation
setupSimulation
setupStart
simulationStart
simulationCompile andplot results
Compile andplot results
Combine all individualsimulation results
Combine all individualsimulation results
Check how well your protocol performed
Check how well your protocol performed
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Combine all individualsimulation results
Combine all individualsimulation resultsCombine all individual
simulation resultsCombine all individual
simulation resultsCombine all individualsimulation results
Combine all individualsimulation results
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 14
Motivation and Essence of the paper in Summary
Now
Simulationsetup
Simulationsetup
Startsimulation
Startsimulation
Check how well your protocol performed
Check how well your protocol performed
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Combination of all simulation results Combination of all simulation results
Automated plottingAutomated plotting
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 15
Agenda
Background on Evaluation Approaches
PeerfactSim.KOM
Requirements on Simulating P2P Systems
Comparative Evaluation in PeerfactSim.KOM
Conclusions
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 16
Approach
1. Common interfaces
2. General analyzers
3. Configurations and automated result compilation
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 17
Analyzers
Collecting– Events are monitored and reported to analyzers– Analyzers maintain overview on metrics
Hierarchy of output on three different layers:
– Comparative evaluation (for all parameter combinations)
– Multiple-seed evaluationfor one parameter setup
– Single evaluation of one setup with one seed
Output style:– ordered by the time– grouped by the single peers– grouped by the single peers
and sorted by their values
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 18
Analyzers by Layers
DefaultAggregationAnalyzer: query events in aggregation nodes
DefaultDHTOverlayAnalyzer: lookup and store events in a DHT overlayDefaultKBROverlayAnalyzer: query and routing events in a KBR overlayDefaultUnstructuredAnalyzer: connections and queries in an unstructured
overlay
DefaultTransAnalyzer: sent and received messages in the transport layer DefaultNetAnalyzer: sent and received messages in the network layerDefaultChurnAnalyzer: session times and inter-session timesDefaultConnectivityAnalyzer: on- and offline events of nodes
DefaultOperationAnalyzer: operations in the simulatorDefaultMemoryUsageAnalyzer: memory used by the simulatorDefaultTimeUsageAnalyzer: time used by the simulator
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 19
Sample Evaluations: All Overlays
Setup: 1000 nodes
Overlays– Unstructured: Gnutella06 and Gia – Structured: Chord, EpiChord, ReChord, CaReChord, Pastry and
CAN
Seeds: 20
Actions:– 120 min joining phase– 60 min publishing phase– 120 min lookup phase– Publish and lookup files from a file set based on a Zipf distribution
Network: Simple network, no churn
Interest: Successful lookups/search, bandwidth consumption
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 20
Successful Downloads
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 21
Bandwidth Consumption
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 22
Sample Evaluations: Structured Overlays
Setup: 1000 nodes
Overlays– Structured: Chord, EpiChord, ReChord, CaReChord, Pastry and CAN
Seeds: 20
Actions:– 120 min joining phase– 180 min lookup phase (1 lookup every 30-120 seconds)– Publish and lookup files from a file set based on a Zipf distribution
Network: Global Network Positioning, no churn
Interest: Hop count, traffic / bandwidth usage
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 23
Hop Count Measure
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 24
Served Messages per Node
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 25
Forwarded Messages
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 26
Bandwidth Consumption
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 27
Sample Evaluations: Chord Variants
Setup: 1000 nodes
Overlays– Structured: Chord, EpiChord, ReChord, CaReChord
Seeds: 20
Actions:– 120 min joining phase– 180 min lookup phase (1 lookup every 30-120 seconds)– Publish and lookup files from a file set based on a Zipf distribution
Network: – Modular network layer: measurement-based delay, packet loss,
jitter and bandwidth.– Churn: based on measurements in KAD by Steiner et al.
Interest: Hop count, traffic / bandwidth usage
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 28
Average Hop Count
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 29
Lookup Duration
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 30
Bandwidth Consumption
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 31
Simulation Time
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 32
Conclusions
Simulation– Common approach to evaluate large scale (p2p) networks
Requirements– Setup / configure once, collect all results at the end– Comparative evaluation of all parameter combinations
Simulationsetup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot resultsSimulation
setup
Simulationsetup
Startsimulation
Startsimulation
Compile andplot results
Compile andplot results
Combine all individualsimulation results
Combine all individualsimulation results
Check how well your protocol performed
Check how well your protocol performed
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Simulationsetup
Simulationsetup
Startsimulation
Startsimulation
Check how well your protocol performed
Check how well your protocol performed
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Combination of all simulation results Combination of all simulation results
Automated plottingAutomated plotting
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.Sim.Sim.
Sim.Sim.NowNow
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 33
Join PeerfactSim.KOM (see www.peerfact.org)
Type– Event-based simulator– Written in Java– Simulations up to 100K peers possible– Focus on simulation of p2p systems on various layers
• User
• Application
• Services: monitoring, replication …
• Overlays
• Network models
Invitation to join the community– Several universities use and extend the simulator actively– Used and heavily extended in the project
Jun.-Prof. Dr.-Ing. Kalman Graffi 13. April 2023 34
Thank You for Your Attention
Jun.-Prof. Dr.-Ing. Kalman Graffi Technology of Social Networks GroupInstitute of Computer ScienceHeinrich-Heine-Universität Düsseldorf
eMail: [email protected] Web: www.p2pframework.comWeb: www.peerfact.org
??
![Cloud Computing Lec7 V1...– Bulk Synchronous Parallel model [Valiant, 1990][Jakovits et al, HPCS 2013] – Building a fault-tolerant BSP framework (NEWT) [Kromonov et al, HPCS 2014]](https://img.dokumen.tips/doc/110x75/5ec97df1b216631abf2785e6/cloud-computing-lec7-v1-a-bulk-synchronous-parallel-model-valiant-1990jakovits.jpg)
