Scalable overlay Networks

15.01.2018

HELSINGIN YLIOPISTOHELSINGFORS UNIVERSITETUNIVERSITY OF HELSINKI

Faculty of SciencesDepartment of Computer Science

1Scalableoverlay networks

Scalable overlay Networks

Dr. Samu Varjonen15.01.2018

15.01.2018




Contents

● Course overview

● Lectures

● Assignments/Exercises

15.01.2018




Course Overview

● Overlay networks and peer-to-peer technologies have become key components for building large scale distributed systems.

● This course will introduce overlay networks and peer-to-peer systems, discuss their general properties, and applications. The course will cover the following topics:

– Currently deployed peer-to-peer systems and how they work

– Distributed Hash Tables as a base for structured peer-to-peer systems

– Peer-to-peer storage systems

– Performance issues, legal aspects, and privacy issues

– Overlay content distribution algorithms

15.01.2018




General Info

● Advanced course, 5 credits

● Requirements: basics of networking

● Assignments/exercises done as individual work.

15.01.2018




Lectures

● Lectures

● Monday 12:15-14:00 in C122 15.1.-26.2.

● Thursday 12:15-14:00 in DK117 18.1-1.3.(The last one if needed)

● Assignments/exercises

– 16.01.-27.02.18 tu 12.15-14.00 Exactum, C323 ● Course based on book

– S. Tarkoma. Overlay Networks: Toward Information Networking. 260 pages. CRC Press / Auerbach, February 2010.

15.01.2018




The course book

Some of the newer material discussed on thecourse are not in the book but I will provide links to papers discussing the topics.

15.01.2018




Exam material: additional articles

● Article: G. DeCandia et al. Dynamo: Amazon’s Highly Available Key-value Store. SOSP 2007.

● Article: Theory and Practice of Bloom Filters for Distributed Systems. IEEE Surveys and Tutorials. 2012.

● Slideset: Modeling and Analysis of Anonymous-Communication Systems. J. Feigenbaum. Presentation at WITS 2008. General idea of MIX and Onion Routing are part of the exam material.

● Article: R. Cohen et al. Resilience of the Internet to Random Breakdowns. Phys. Rev. Lett. 85, 4626–4628 (2000). Background info and derivation of the resiliency formula (power law lectures). Additional details not part of exam material.

● Article: D. Kreutz, F. M. V. Ramos, P. Verissimo, C. E. Rothenberg, S. Azodolmolky, S. Uhlig. Software-defined Networking: A Comprehensive Survey. Sections I,II, V and VI are part of the exam material. Proceedings of the IEEE, Vol. 103, Issue 1, Jan. 2015.

15.01.2018




Lectures● MO 15.01. C122 Introduction. Exercises. Motivation.

● TH 18.01. DK117 Unstructured networks I

● MO 22.01. C122 Unstructured networks II

● TH 25.01. DK117 Bittorrent and evaluation

● MO 29.01. C122 Privacy (Freenet etc.) and intro to power-law networks.

● TH 01.02. DK117 Consistent hashing. Distributed Hash Tables (DHTs)

● MO 05.02. C122 DHTs continued

● TH 08.02. DK117 Power-law networks

● MO 12.02. C122 Power-law networks and applications.

● TH 15.02. DK117 Applications I

● MO 19.02. C122 Applications I

● TH 22.02. DK117 Advanced topics

● MO 26.02. C122 Conclusions and summary

● TH 01.03. DK117 Reserved

15.01.2018




Grading

● Course grading will be based on the final exam and the assignments/exercises.

● Course exam 7.3.2018 16:00 B123

● Separate exam on 25.4.2018 16:00 in B123

● You can use the exercise score with the first separate exam

15.01.2018




Assignments/Exercises

● Assignments are optional and will grant you extra points towards the exam.

● Assignments are marked and presented in the answers sessions every two weeks.

● You have to attend the session to get the points.

● Max exam points 18, max bonus points 4

● Tuesdays 12:15-14:00 C323 last session 27.2.

● 23.1. Reception on questions I

● 30.1. Answers due to questions I

● 6.2. Reception on questions I

● 13.2. Answers due questions II

● 20.2. Reception questions II

● 27.2. Answers due to questions II

15.01.2018




Main theme Prerequisites Approaches learning goals Meets learning goals Deepens learning goals

Overlay and peer-to-peer networks: definitions and systems

Basics of data communications and distributed systems (Introduction to Data Communications, Distributed Systems)

Knowledge of how to define the concepts of overlay and peer-to-peer networks, and state their central features

Ability to describe at least one system in detail

Ability of being able to compare different overlay and p2p networks in a qualitative manner

Ability to assess the suitability of different systems to different use cases

Ability to give one’s own definition of the central concepts and discuss the key design and deployment issues

Distributed hash tables

Basics of data communications and distributed systems (Introduction to Data Communications, Distributed Systems)Big-O-notation and basics of algorithmic complexity

Knowledge of the concepts of structured and unstructured networks and the ability to classify solutions into these two categoriesKnowledge of the basics of distributed hash tablesAbility to describe at least one distributed hash table algorithm in detail

Ability of being able to compare different distributed hash table algorithmsAbility of designing distributed hash table-based applicationsKnowledge of key performance issues of distributed hash table systems and the ability to analyze these systems

The knowledge of choosing a suitable distributed hash table design for a problemFamiliarity with the state of the art

Reliability and performance modelling

Basics of probability theoryBasics of reliability in distributed systems

Ability to model and assess the reliability of overlay and peer-to-peer networks by using probability theoryKnowledge of the most important factors pertaining to reliability

Ability of analytically analyzing the reliability and performance of overlay and peer-to-peer networksUnderstanding of the design issues that are pertinent for reliable systems

Familiarity with the state of the art

Content distribution Introduction to Data Communications

Knowledge of the basic content distribution solutionsAbility to describe at least one overlay and p2p network based content distribution solution

Knowledge of different content distribution systes and the ability to compare them in detailKnowledge of several content distribution techniques

Familiarity with the state of the art

Security Basics of computer security Knowledge of the basic security issues with overlay and p2p networksKnowledge of the sybil attack concept

Ability to discuss how security problems and limitations can be solvedKnowledge of how to prevent sybil attacks

Knowledge of how to prevent sybil attacksFamiliarity with the state of the art

15.01.2018




Contact information

● Lecturer PhD Samu Varjonen (@cs.helsinki.fi)

● Assignments: Juhani Toivonen (@cs.helsinki.fi)

● Course homepage can be found: https://courses.helsinki.fi/fi/csm13104/

15.01.2018




Introductionand unstructured networks

15.01.2018




Introductionand unstructured networks

● Overlay networks and intro to networking

● Unstructured networks

15.01.2018




Overlay networks

● An overlay network is a network that is built on top of an existing network.

● The overlay therefore relies on the so called underlay network for basic networking functions, namely routing and forwarding.

● Today, most overlay networks are built in the application layer on top of the TCP/IP networking suite.

Internet (TCP/IP)Internet (TCP/IP)

ClusterCluster Wide-area (unstructured)Wide-area (unstructured)

SearchSearch StorageStorage RendezvousRendezvousSearchSearch StorageStorage RendezvousRendezvous

Wide-area (structured)Wide-area (structured)

SearchSearch StorageStorage RendezvousRendezvous

DHT

Examples:Lookup: Chord, CAN, Kademlia

Storage: PASTRendezvous: Scribe (for

multicast), i3

Good for name/value data, note flat address space, one node is responsible, churn is a concern

Good for name/value data, note flat address space, one node is responsible, churn is a concern

Limited flooding / depth first / Bloom

filters

Examples: Gnutella and

Freenet

Tracker

Example: BitTorrent

Good for arbitrary data and search functions, can aggregate routing info, structure improves

scalability

Good for arbitrary data and search functions, can aggregate routing info, structure improves

scalability

Consistent hash (O(1) DHT)

Selective flooding

Replication, Gossip, etc.

Examples: Dynamo, Cassandra

Consistent hashing alleviates network problems and eventual consistency can be achieved

through replication and synchronization

Consistent hashing alleviates network problems and eventual consistency can be achieved

through replication and synchronization

15.01.2018




Overlay networks

● Overlay technologies can be used to overcome some of the limitations of the underlay, and at the same time offering new routing and forwarding features without changing the routers.

● The nodes in an overlay network are connected via logical links that can span many physical links. A link between two overlay nodes may take several hops in the underlying network.

15.01.2018




Requirements for overlay networks

1) Support the execution of one or more distributed applications by providing infrastructure for them.

2) Participate and support high-level routing and forwarding tasks. The overlay is expected to provide data forwarding capabilities that are different from those that are part of the basic Internet.

3) Deployed across the Internet in such a way that third parties can participate in the organization and operation of the overlay network.

15.01.2018




Protocol stack

● Layers are part of a network architecture

– Provide services for layers above

– Hiding the complexity of the current layer● Multiple layers are needed in order to reduce

complexity

– Separation of network functions

– Distribution of complexity

– OSI, TCP/IP● Protocols are building blocks of a network design

– Can exist independently of layering

15.01.2018




Background

● What is network architecture?

● Layered architecture

● Network structure

● Network evolution

● Motivation for overlay networks

15.01.2018




Network architecture

● A set of principles and basic mechanisms that

● guide network engineering

– Physical links

– Communication protocols● Format of messages● The way in messages are exchanged● Elements of the protocol stack● Where is the state?

15.01.2018




Naming, addressing, and routing

NamingHow to identify and name a node? Even if its address changes.

AddressingWhere is the node located?

RoutingHow to route information to the node’s address?

unicast: to a specific nodebroadcast: to all nodesmulticast: to a subset of nodesanycast: to any one in some subset (IPv6)

15.01.2018




Underlying network (link layer, physical layer)

Underlying network (link layer, physical layer)

TCP/IP network stack

Networking layer (IP)

Networking layer (IP)

Transport layer (TCP / UDP)

Transport layer (TCP / UDP)

Application layer (FTP, SMTP, etc.)Application layer (FTP, SMTP, etc.)

15.01.2018




The hourglassDiverse applications

Divergence

Transport layer (TCP/IP)

Convergence

Diverse physical layers

15.01.2018




Structure of the Internet

● End-to-End Principle: application logic and state at the edge, core network is simple

● Network maintains routing tables and forwards packets based on IP prefixes

● Various protocols for intra-domain routing (OSPF, ...)

● Hierarchical structure: Autonomous Systems with tiers and peering links, economics of inter-domain routing

● Border Gateway Protocol (BGP) for routing between domains

● New requirements for IP and network: mobility, multihoming, security, multipath operation, ...

15.01.2018




Internet evolution

● IPv6

● Caching

● CDNs

● Overlays

● Cloud computing and datacenter networking

● Software-defined Networking (SDN)

– SDN allows to introduce new concepts in the network

– Push down overlay technology into the network layer● 5G Radio and Core Network

15.01.2018




Questions

● Q: Where is the routing state in overlay networks?

● A: Must be at the edge of the network. Easy to bootstrap, because network is not modified.

● Q: What is the benefit of this from the network point of view?

● A: Can overcome limitations of the current network and introduce new services.

● Q: What is limitation from the network point of view?

● A: Must take network structure into account (economics). Traversal solutions can be complex.

Documents

Scalable overlay Networks