Peer-to-peer Virtual Private Networks and Applications

Renato Jansen Figueiredo

Associate Professor

Cloud and Autonomic Computing Center/ACIS Lab

University of Florida

Visiting Researcher at VU

2

Backdrop

Virtual machines in cloud computing

On-demand, pay-per-use, user-configurable

Federated environments

End-to-end Internet connectivity hindered by address

space and presence of NATs, firewalls

Network virtualization – seamlessly connecting

virtual machines across multiple providers

3

Rationale

Virtualization techniques for decoupling,

isolation, multiplexing also apply to networking

E.g. VLANs, VPNs

However, there are challenges in configuration,

deployment, and management

Peer-to-peer techniques provide a basis for

scalable routing, and self-management

Software routers, integration at network end-points

enables deployment over existing infrastructure

Architecture, design needs to account for

connectivity constraints, and support TCP/IP

efficiently; optimize for common cases

Application Examples

Cloud-bursting

Run additional worker VMs on a cloud provider

Extending enterprise LAN to cloud VMs – seamless

scheduling, data transfers

Federated “Inter-cloud” environments

Multiple private clouds across various institutions

Virtual machines can be deployed on different sites

and form a distributed virtual private cluster

Connecting devices of social network peers

Media streaming, file sharing, gaming, …

4

5

Talk - Outlook

Background

Architecting self-organizing virtual networks

Topology, routing, tunneling, addressing, NAT

traversal, performance

Uses in Grid/cloud and end-user environments

Virtual Private Clusters

Social VPNs

Applications

FutureGrid – high-throughput computing virtual

appliances

ConPaaS

6

Resource Virtualization

Virtual machines (Xen, VMware, KVM) paved

the way to Infrastructure-as-a-Service (IaaS)

Computing environment decoupled from physical

infrastructure

Pay-as-you-go for computing cycles

Virtual networks complement virtual machines

for increased flexibility and isolation in IaaS

VMs must communicate seamlessly – regardless of

where they are provisioned

Traffic isolation; security, resource control

7

VMM +

VN

Virtual Machines and Networks

WAN

Domain A Domain C

Domain B

V1

V2

V3

Physical

Infrastructure

Virtual

Infrastructure

Virtual Networks

Single infrastructure, many virtual networks

E.g. one per user, application, project, social

network…

Each isolated and independently configured

Addressing, protocols; authentication, encryption

Multiplexing physical network resources

Network interfaces, links, switches, routers

8

Network Virtualization – Where?

9

Network Device

Network Device

Network Fabric

(Virtual) machine

(Virtual) machine

Software Software Virtualized endpoints

Virtualized Fabric (e.g VLAN, OpenSwitch)

10

Landscape

Peer-wise Internet connectivity constrained

IPv4 address space limitations; NATs, firewalls

Challenges - shared environment

Lack of control of networking resources

Cannot program routers, switches

Public networks – privacy is important

Often, lack privileged access to underlying resources

May be “root” within a VM, but lacking hypervisor privileges

Dynamic creation, configuration and tear-down

Complexity of management

Technologies and Techniques

Amazon VPC:

Virtual private network extending from enterprise to resources at

a major IaaS commercial cloud

OpenFlow:

Open switching specification allowing programmable network

devices through a forwarding instruction set

OpenStack Quantum:

Virtual private networking within a private cloud offered by a

major open-source IaaS stack

ViNe:

Inter-cloud, high-performance user-level managed virtual

network

IP-over-P2P (IPOP)

Peer-to-peer, inter-cloud, self-organizing virtual network

11

Example: OpenFlow

Towards an open platform foundation

supporting “Software-Defined Networks” (SDN)

Interface standardized by Open Networking

Foundation (ONF)

Board members (as of 6/12): Google, Microsoft,

Facebook, Yahoo!, Deutsche Telekom, NTT, Verizon

Dozens of members: Citrix, Huawei, Orange, IBM,

Dell, HP, Oracle, Goldman-Sachs, …

Current (as of 6/12): OpenFlow Switch 1.3.0

12

OpenFlow Switch and Controller

13

Controller

Secure Channel

Group Table

Flow Table

Flow Table Pipeline

OpenFlow Protocol

Add, update, delete

Match flow table entry

Table miss

OpenFlow ingress port

OpenFlow output port

14

Peer-to-Peer Virtual Networks

Overview

User-level IP overlays deployable on Internet end

resources (software routers, virtual NICs)

Why virtual?

Hide complexities associated with NAT traversal,

IPv4 address space constraints from applications

Support unmodified applications

Why peer-to-peer?

Self-organizing - reduce management complexity

and cost

Decentralized architecture for scalability and

robustness

15

The IP-over-P2P (IPOP) Approach

Isolation Virtual address space decoupled from Internet

Packets picked, encapsulated, tunneled and delivered within the scope of virtual network

Self-organization Overlay topology, routing tables

Autonomously deals with joins, leaves, failures

Decentralized P2P messaging architecture No global state, no central point of failure

Tunnels (UDP, TCP, …), routing

Decentralized NAT traversal No need for STUN server infrastructure

[IPDPS 2006, Ganguly et al]

Application

VNIC

Virtual Router

Virtual Router

VNIC

Application Wide-area Overlay network

Isolated, private virtual address space

10.10.1.2

10.10.1.1

Unmodified applications Connect(10.10.1.2,80)

Capture/tunnel, scalable, resilient, self-configuring Routing; object store

IPOP: Architecture Overview

n1 n7

Bi-directional ring ordered by 160-bit IPOPid’s

Structured connections:

• “Near”: with neighbors

• “Far”: across the ring

Near

Far

Multi-hop path

between n1 and n7

n5

n6 n2

n3 n4

n8

n9

n10

n11

n12

n1 < n2 < n3 <……. < n13 < n14

P2P Overlay (Brunet)

IPOPid

18

Overlay: Edges and Routing

Overlay edges

Multiple transports: UDP, TCP, TLS

NAT traversal (UDP hole-punching)

Greedy routing

Deliver to peer closest to destination IPOPid

Constant # of edges per node (average k)

O((1/k)log2(n)) overlay hops

On-demand edges

Created/trimmed down based on IP communication

19

Creating Overlay Edges

• A sends a Connect-to-me (CTM) request to B’s IPOPid

• Contains all its URIs (UDP/TCP IP:port endpoints)

• Routed over P2P overlay to B

A B

CTM request

Overlay path

CTM reply

• B sends CTM reply with its URIs – overlay routed

Link request

• B initiates “linking” with A

• Attempts linking with parallel requests to A’s URIs

A’s endpoint URIs: tcp://10.5.1.1:3000 (local) udp://128.227.56.9:4433 (NAT – learned)

B’s endpoint URIs: tcp://172.16.2.1:5000 udp://129.15.56.9:6000

20

NAT Traversal

A B

Direct edge between A

and B

Technique for “cone” UDP NATs: A’s link request message to B creates ephemeral state in A’s NAT allowing messages from B to pass through NAT (and vice-versa) Overlay: manage keep-alives so NAT mapping “holes” stay open; re-link if NAT mappings expire

21

Naming and Multiplexing

One P2P overlay can multiplex multiple VNs E.g. multiple virtual clusters from different projects

IP routing within the scope of a namespace

User-provided string identifies IPOP namespace Each IPOP node is configured with a namespace

IP-to-P2P address resolution: DHT-Get(namespace:IP) -> IPOPid

22

Managing Virtual IP Addresses

Address assignment: static, or dynamic Supports DHCP

Store configuration (including base address, mask) on DHT entry bound to namespace

DHCP proxy runs on each IPOP node Pick DHCP request

Lookup DHCP configuration for namespace

Guess an IP address at random within range

Attempt to store in DHT; wait for majority to acknowledge; retry upon failure

23

At each node:

Count IP-over-P2P packets to other nodes

When number of packets within an interval

exceeds threshold:

Initiate connection setup; create edge

Trimming on-demand edges no longer in use

Overhead involved in connection maintenance

Optimization: On-demand edges

24

Optimization: Tunnel Edges

Peers X, Y may not be able to communicate

directly if they are behind symmetric NATs

X, Y exchange list of neighbor URIs

Each attempts to create edge to common

intermediary Z to serve as proxy

Routing – abstracted as regular overlay edge

X-Y connected by virtual edge

Useful to maintain ring topology in the face of failures

(routing outages, symmetric NATs)

25

Implementation

IPOP open-source system

C# user-level router

Tap virtual network device

Performance

1GbE physical LAN

Latency (ms) Bwidth (Mb/s) Mem (KB)

Host 0.27 941 n/a

IPOP 0.52 284 38312

IPOP+sec 0.75 55 50976

Performance (WAN)

EC2/UF EC2GoGrid UF/GoGrid

Netperf stream

native (Mbps)

89.2 35.9 30.2

Netperf stream

IPOP (Mbps)

75.3 19.2 25.7

Netperf RR

trans/s native

13.4 11.1 10.0

Netperf RR

trans/s IPOP

13.3 10.7 9.8

27

IPOP provides core primitives for packet

capture/injection and overlay routing

How to control which nodes connect to a

particular IP namespace?

Focus on two approaches:

Each peer decides the peers they connect with –

SocialVPN

Peers join groups and agree on a trusted third party

as mediator – GroupVPN

Access Control

28

Users now commonly manage relationships to

social peers through Online Social Networks

Facebook, Google+

Communication hindered by OSN provider

APIs, privacy concerns

A generic IP network can enable existing and

new social network applications

But users don’t have public IPs, don’t want to

necessarily open NATs/firewalls to all users

Users don’t want to configure and discover network

services manually

SocialVPN

Bob Alice

Alice's Compute Node Alice's Friend's Compute Node

Carl

Bob's Compute

Node on EC2

Social VPNs

OSN

XMPP

IP-over-P2P Tunnel

30

Social VPNs

From a user’s perspective: it’s simple

My computer gets a virtual network card

It connects me directly to my social peers

All IP packets: authenticated, encrypted, end-to-end

Leverage well-known PKI techniques

No configuration besides establishing social links

All I need to do to is log in to a web based social network

Applications, middleware work as if the

computers were on the same local-area network

Including multicast-based resource discovery –

UPnP, mDNS

31

Applications

Social VPN is not the application

It is not tied to an application either

It enables applications that are of interest for collaboration

Security – needed beyond network layer

Authenticated end-to-end private IP tunnels provide a

foundation

Traditional applications

Media streaming, desktop sharing, file sharing, cycle

sharing

Platform for decentralized social network

applications

Fault-tolerant micro-blogging, private file sharing, ..

32

Social VPN Internals

IPOP

NAT traversal and routing core

Private end-to-end tunnels

Peer discovery and certificate exchange

XMPP – Jabber, Google

Facebook APIs (was in first prototype; no longer in the code)

Dynamic IP address assignment

Facebook: more users than IPv4 24-bit private space

Also must avoid conflicts with local private networks, and

support mobility

Addressing and Mapping

160-bit P2P IDs used for overlay routing

Each node generates random P2P ID

Node issues a self-signed public key certificate with

its P2P identifier; publishes through OSN APIs

Certificates of friends’ nodes are discovered,

retrieved, revoked through OSN APIs

IPv4 addresses seen by applications

Dynamically-generated non-conflicting private subnet

Local node and friends’ nodes are mapped

dynamically to addresses within range

Naming possible through SocialDNS

IP src/dest addresses translated (ports are not)

[COPS 2008]

Alice

Alice's Compute Node Alice's Friend's Compute Node Bob's Compute

Node on EC2

Address Translation

Send-to BobP2P

SVPN: 192.168.0.0/16 Alice: 192.168.0.1 Bob: 192.168.2.2 -> BobP2P

SVPN: 172.16.0.0/16 Bob: 172.16.0.1 Alice: 172.16.3.4 -> AliceP2P

0.1 2.2 3.4 0.1 Recv-from AliceP2P

35

Group-oriented VPNs

Well-suited for collaborative environments for

cluster computing

Nodes who join a group have peer-wise

connectivity to all other nodes

Based on public key cryptography

Owner of a group is a certificate authority signing

GroupVPN certificates

Centralized Web-based interface hides low-

level management from users

Users create groups, determine who can join group

Certificate signing automated; group membership

Certificate revocation lists disseminated via P2P

36

Grid appliance clusters

Virtual appliances

Encapsulate software environment in image

Virtual disk file(s) and virtual hardware configuration

The Grid appliance

Encapsulates cluster software environments

Current examples: Condor, MPI, Hadoop

Homogeneous images at each node

IPOP/GroupVPN connecting nodes forms a cluster

Deploy within or across domains

37

Grid appliance - virtual clusters

Same image, per-group VPNs

copy

instantiate

Condor

+

Virtual

Network A Condor worker Another Condor worker

Repeat…

Virtual

machine

Group

VPN

GroupVPN

Credentials

(from

Web site) Virtual IP - DHCP

10.10.1.1 Virtual IP - DHCP

10.10.1.2

38

Grid appliance configuration

At the end of GroupVPN initialization: Each node of a private virtual cluster gets a DHCP

address on virtual tap interface

A barebones cluster

Additional configuration required depending on middleware Which node is the Condor negotiator? Hadoop front-end?

Which nodes are in the MPI ring?

Leverage P2P/IPOP primitives: Distributed hash table

Advertise (put namespace,managerIP); discover (get namespace)

IP multicast discovery over GroupVPN

39

Applications in FutureGrid

FutureGrid testbed

DAS-like system distributed across US institutions

Research, education, development, testing of Grid

and cloud computing middleware, applications

IaaS partitions: Nimbus, OpenStack, Eucalyptus

Virtual networks: ViNe and IPOP

Virtual appliances

Lower barrier to entry – pre-configured environments

40

IPOP + ConPaaS at VU

ConPaaS – framework/runtime to manage platform-as-

a-service environments

Examples: Web service, task farming service

Build upon IaaS primitives to create VMs

Integration with IPOP

Allow deployments to span across multiple providers

(federation; bursting; fault-tolerance)

Within VMs - no changes to IaaS stack

Isolate “data plane” communications from public Internet

Thilo Kielmann, Guillaume Pierre, Contrail/ConPaaS teams

41

IPOP + ConPaaS

WAN Private

Cloud

EC2

Zone

DAS

site

N1

N2

N3

IaaS

Providers

ConPaaS applications,

IPOP namespaces

172.16.10.10 172.16.10.20

172.16.10.20

PlanetLab bootstrap overlays

Grid appliance deployments:

Archer - ~700-CPU cluster

SocialVPN deployments:

Thousands of downloads, hundreds of

deployed nodes

Deployed Systems

43

Integration of IPOP with IPsec for dynamically-

provisioned cloud virtual networks

Contrail, ConPaaS

Overlay by-pass, integration with OpenFlow

software-defined networks

IPv6/IPv4 overlays, virtual clusters for high-

throughput computing, education

Archer (computer architecture)

FutureGrid (virtual appliances for education)

PRAGMA (Pacific Rim Grid)

Unstructured P2P SocialVPN

Discover, bootstrap, route through friends

On-going Work

44

Acknowledgments

ACIS P2P group (IPOP) Over the years: P. O. Boykin, Heungsik Eom, Arijit Ganguly,

Pierre St. Juste, Kyungyong Lee, Yonggang Liu, Girish

Venkatasubramanian, David Wolinsky, Jiangyan Xu

Vrije Universiteit,

ConPaaS team

FutureGrid, National Science Foundation

Awards 0751112, 0758596, 0910812

45

Thank you

For more information and downloads:

http://ipop-project.org

http://socialvpn.org

http://futuregrid.org

http://grid-appliance.org

46

Related Work

There exist several VPN technologies:

Enterprise VPNs (e.g. Cisco); Open-source (e.g

OpenVPN); Consumer/gaming/SMB (e.g. Hamachi)

Not easily applicable to federating cloud resources

Proprietary code; difficulty in configuration/management

Research work in the context of Grid/cloud

computing

VNET (Northwestern University), VIOLIN (Purdue

University), Private Virtual Cluster (INRIA), ViNe

(Tsugawa, Fortes @ UF)

Smartsockets @ VU

47

48

Bootstrapping a New Node

MyIPOPid

Forms a “leaf” connection with a public node - forwarder

Selected at random from list of “bootstrap” nodes

Sends CTM request addressed to its own IPOPid

Received by nearest neighbors

Creates structured connections; trims leaf connection

Forwarder

CTM (MyIPOPid)

Received by left and

right neighbors

(c) Renato Figueiredo 49

B2

IPOP Namespaces

N1

Namespace N1: 10.128.0.0/255.192.0.0

A1: 10.129.6.83

C1

B1: 10.129.6.71

D1

N1:10.129.6.71→

IPOPid x1

x1 x2

x4

x3

x5 x6 x7

x8

N2

A2

C2 D2

DHTLookup(N1:B1)

x1 IPOPid

“ARP cache”

IPOP packet DHTCreate(N2, 10.128.0.0/255.192.0.0)

N2:10.129.6.71→

IPOPid x2

DHTCreate(N2:A2,x2)

50

Social DNS

Motivation: • Users cannot define domain names used to access

their services in VPN settings

• Dynamic private networks; difficult to keep track of services by IP addresses

Objective: • A decentralized, naming service that gives

individuals the ability to select and share the domain names for their resources with SocialVPN peers

Approach: • Short names within social context

• Decentralized architecture where each node runs a local DNS server and communicates via SocialVPN

• Rank-based name conflict resolution

51

Security

End-to-end authentication and encryption IPsec tunneling over IPOP

Reuse existing software stack

End-to-end security implemented in IPOP RSA priv/pub keys

X.509 certificates

Point-to-point authentication and encryption TLS edges have been implemented

Difficulty to deal with NAT traversal

Point-to-point security in IPOP: ongoing work Reuses framework and code base from end-to-end

Avoid double-traversal of security stack for performance (e.g. shortcut connections based on IP traffic inspection)

52

Security

Appliance firewall Block outgoing packets to physical net

Except DHCP, DNS, IPOP’s UDP port

Confine traffic to within WOW and host-only

IPsec or IPOP security With IPsec, kernel/user tools reused – unmodified

Network routing is P2P, however: Trust can be managed by central CA

All intra-WOW communication authenticated and end-to-end encrypted using X.509-based PKI

Private net/netmask 10 lines of IPsec configuration for entire WOW

53

Linking and NAT traversal

Src = R:A

Dst = N:Y

Src = N:Y

Dst = M:X

R:A M:X N:Y S:B

Outgoing packet to M:X

(hole punched)

Src = S:B

Dst = M:X

Src = M:X

Dst = N:Y

Outgoing packet to N:Y

(hole punched)

Src = M:X

Dst = S:B

Exchange each other’s NAT assigned IP:port

M:X N:Y

NAT N 128.139.156.90

NAT M 128.227.56.83

S:B R:A Allow

Dropped

VNIC

Virtual Router

Virtual Router

VNIC

Application

Wide-area Overlay network

Local Interface

LAN Router

NIC

Application

NIC

Application

Avoiding LAN overheads

Supporting IPOP Routers

● Single IPOP router for a

(V)LAN

● Avoid need for IPOP

software stack on end

host

● Avoid IPOP overhead on

LAN communication

IP=10.1.1.2

Eth=A:B:C:D:E:0

IP=10.1.1.3

Eth=A:B:C:D:E:1

IP=10.1.1.4

Eth=A:B:C:D:E:2

TAP

Device

VPN

SoftwareNIC0

NIC1

Virtual Router

Internet

DHCP

● Provides address allocation and DNS settings

● IPOP router keeps a history of allocations and

ignores packets destined for them sent within the

(V)LAN

IP=10.1.1.2

Eth=A:B:C:D:E:0

IP=10.1.1.3

Eth=A:B:C:D:E:1

IP=10.1.1.4

Eth=A:B:C:D:E:2

TAP

Device

VPN

SoftwareNIC0

NIC1

Virtual Router

Internet

DHT

DHCP request

ARP

● Lookup Ethernet address from IP address

● IPOP router ignores ARP if IP in (V)LAN

● If destination is not on the LAN, check if such a

peer exists in the overlay

● Reply IPOP router addr.

IP=10.1.1.2

Eth=A:B:C:D:E:0

IP=10.1.1.3

Eth=A:B:C:D:E:1

IP=10.1.1.4

Eth=A:B:C:D:E:2

TAP

Device

VPN

SoftwareNIC0

NIC1

Virtual Router

Internet

ARP

not in router table? DHT

Local reply

Connectivity

Each node is given an IP

address and domain name

Access Control

The user locally decides to

allow or block another user

Trust

Use current social networking

systems (XMPP) to bootstrap

secure connections with

friends

Social VPN Prototype

Bob Alice

Alice's Compute Node Alice's Friend's Compute Node

Better QoS by leveraging

proximity or social trust

Flexibility in selecting

from a collection of

trusted compute nodes Alice can leverage

Her own resources

to add more

computational

power to her device

and save energy

Carl

Bob's Compute

Node on EC2

Cloud provider is now just

one more compute node

Computation offloading

Example: Resource discovery

Service discovery time

100 UPnP servers over WAN

U. Chicago, UC San Diego, and U. Texas

UPnP client located at U. Florida

Servers connected to PlanetLab SocialVPN overlay

Resource Discovery

Service discovery time

U. Texas U. Chicago UC San Diego

Service discovery time

(ms)

min max min max min max

27.0 29.4 45.4 47.6 54.6 57.0

• SocialVPN supports

unmodified UPnP applications

with service discovery time

commensurable to WAN

latency

• Wi-Fi setup has longer service

discovery time than wired LAN

(figure)

Offloading

Offloading to PC and EC2

Energy consumption

• The benefits are compelling at large image sizes

• Higher power consumption of offloading to Amazon EC2 than

offloading to local workstation due to network latency

Grid Appliance / Archer

2. Boot appliance:

automatically

joins Archer pool

Free pre-packaged Archer

Virtual appliances - run

on free VMMs (VMware,

VirtualBox, KVM)

Portal and Wiki:

Community-contributed

content: applications,

datasets, tutorials

Archer seed resources

300+ cores – Fall 2008

Archer Global

Virtual Network

System software:

Condor scheduler

NFS file systems

Simulators:

Simics, SESC,

Simplescalar

1: Download

appliance

3. Run architecture

simulation jobs on the

Archer pool through Condor

2. Boot appliance:

automatically

joins Archer pool

Free pre-packaged Archer

Virtual appliances - run

on free VMMs (VMware,

VirtualBox, KVM)

Portal and Wiki:

Community-contributed

content: applications,

datasets, tutorials

Archer seed resources

300+ cores – Fall 2008

Archer Global

Virtual Network

System software:

Condor scheduler

NFS file systems

Simulators:

Simics, SESC,

Simplescalar

1: Download

appliance

3. Run architecture

simulation jobs on the

Archer pool through Condor