Adaptive Virtual Networking For Virtual Machine-based Distributed Computing Peter A. Dinda Prescience Lab Department of Computer Science Northwestern University

Adaptive Virtual Networking For Virtual Machine-based

Distributed Computing

Peter A. DindaPrescience Lab

Department of Computer Science

Northwestern University

http://plab.cs.northwestern.edu

2

People and Acknowledgements

• Students– Ashish Gupta, Ananth Sundararaj, Alex

Shoykhet, Jack Lange

• Collaborators– In-Vigo project at University of Florida

• Renato Figueiredo, Jose Fortes

• Funders/Gifts– NSF through several awards, VMWare

3

IBM xSeriesvirtual cluster(64 CPUs),1 TB RAID

Northwestern

Internet

Sun Enterprise servers(E450, E250; 6 CPUs)

Development cluster(5 PowerEdge, 10 CPUs)

IBM xSeriesVirtual cluster(64 CPUs)

GbE switch

IBM zSeries mainframe(1-way, 3.36TB storage)

RAID array(1.2TB)

UFL

10/100 switch

IBM xSeriesDev. cluster(8 CPUs)

InteractivityEnvironmentCluster, CAVE(~90 CPUs),8 TB RAID

2 DistributedOptical TestbedClustersIBM xSeries (14-28 CPUs),1 TB RAID

Nortel OpteraMetro EdgeOptical Router

Distributed Optical Testbed(DOT) Private Optical Network

DOT clusters with opticalconnectivityIBM xSeries (14-28 CPUs),1 TB RAID: Argonne, U.Chicago, IIT, NCSA, others

4

Users already know how to deal with this complexity at another level

5

Virtuoso: “The Dell Model”

A. Shoykhet, J. Lange, and P. Dinda, Virtuoso: A System For Virtual Machine Marketplaces, Technical Report NWU-CS-04-39, July, 2004.

R. Figueiredo, P. Dinda, J. Fortes, A Case For Grid Computing on Virtual Machines, Proceedings of the 23rd International Conference on Distributed Computing Systems (ICDCS 2003)

6

The Illusion

User

User’s LAN

VM

Your machines are sitting next to you.

7

Virtual Machines• Language-oriented VMs

– Abstract interpreted machine, JIT Compiler, large library– Examples: UCSD p-system, Java VM, .NET VM

• Application-oriented VMs– Redirect library calls to appropriate place– Examples: Entropia VM

• Virtual servers– Kernel makes it appear that a group of processes are running on a separate

instance of the kernel or run OS at user-level on top of itself– Examples: Ensim, Virtuozzo, UML, VServer, FreeVSD …

• Microkernels designed to host OSes– Xeno VM

• Virtual machine monitors (VMMs)– Raw machine is the abstraction– VM represented by a single image– Examples: IBM’s VM, VMWare, Virtual PC/Server, Plex/86, SIMICS,

Hypervisor, DesQView/TaskView. VM/386

8

Claim

• Virtual networking for VMs enables the broad application of dream techniques…– Adaptation– Resource reservation

• … using existing, unmodified applications and operating systems– So actual people can use the techniques

9

Questions

• Is there enough application information?– Resource demands– Goals

• Is there enough resource information?– Cycles– Bandwidth

• Are there sufficient adaptation and reservation mechanisms?

• Is the control loop fast enough?

10

Outline• Motivation and claims

• VNET: A virtual network for virtual machines– And what it enables

• VTTIF: Application topology inference• Dynamic topology adaptation

– Combining VNET and VTTIF

• Current directions

• Conclusions

11

Why Virtual Networking?

• A machine is suddenly plugged into your network. What happens?– Does it get an IP address?– Is it a routeable address?– Does firewall let its traffic through?– To any port?

How do we make virtual machine hostileenvironments as friendly as the user’s LAN?

12

VNET: A Layer 2 Virtual Network for the User’s Virtual Machines

• Why Layer 2?– Protocol agnostic– Mobility– Simple to understand – Ubiquity of Ethernet on end-systems

• What about scaling?– Number of VMs limited (1024/user)– Hierarchical routing possible because MAC

addresses can be assigned hierarchically

A. Sundararaj, P. Dinda, Towards Virtual Networks for Virtual Machine Grid Computing, USENIX VM 2004

13

A Simple Layer 2 Virtual Network

Client Server

Remote VM

PhysicalNIC

VM monitor

VirtualNIC

PhysicalNIC

SSH

Hostile Remote NetworkFriendly Local Network

14


Client Server

Remote VM

PhysicalNIC

VM monitor

VirtualNIC

PhysicalNIC

SSH


15


Client Server

Remote VM

PhysicalNIC

VM monitorvnetd vnetd

VirtualNIC

PhysicalNIC

UDP, TCP, TCP/SSL, orSSH tunnel


16

More Details

Host

VM

ProxyVNET

Client

vmnet0ethx

ethz “eth0”

VNET

ethy“eth0”

ClientLAN IP Network

Ethernet Packet Tunneledover TCP/SSL Connection

Ethernet Packet Captured by PromiscuousPacket Filter

Ethernet Packet Injected

Directly into VM interface

“Host Only” Network

VNET 0.9 available from http://virtuoso.cs.northwestern.edu

A collection of such Proxy/Host connections forms a star network centered at the Proxy on the user’s network

17

Initial Performance Results (LAN)

0

2

4

6

8

10

12

Faster than NAT approachLots of room for improvementThis version you can download and use right now

18

VNET 1.0: Bootstrapping the Virtual Network

• Star topology always possible• Topology may change

• Links can be added or removed on demand• Virtual machines can migrate

• Forwarding rules can change• Forwarding rules can be added or removed on demand

Host + VNETd

Proxy + VNETd

VM

19

Current Status SnapshotsPseudo proxy

20

VNET 1.0 Performance

• BW and latency similar to VNET 0.9

• Add/Delete Link: 21 ms

• Add/Delete Rule: 16 ms

• IBM e1350 cluster, 100 mbit switch

21

VNET 1.0 Topology Manipulation(Eight VMs)

0

0.5

1

1.5

2

2.5

3

3.5T

ime

in S

econ

ds

AlltoAll

Bus

Ring

Mesh

Setup Teardown

22

VNET 1.0 Topology Manipulation (Eight VMs)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Tim

e in

Sec

onds

AlltoAll - Bus

AlltoAll - Ring

AlltoAll - Mesh

Bus - Ring

Bus - Mesh

Ring - Mesh

Switch Forward Switch Reverse

23

VNET 1.0 Topology Manipulation (Eight VMs)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Tim

e in

Sec

onds

AlltoAll - Bus

AlltoAll - Ring

AlltoAll - Mesh

Bus - Ring

Bus - Mesh

Ring - Mesh

Switch Forward Switch Reverse

24






• Conclusions

25

VMLayer

VnetdLayer

PhysicalLayer

26

VMLayer

VnetdLayer

PhysicalLayer

Application communicationtopology and traffic load;application processor load

27

VMLayer

VnetdLayer

PhysicalLayer


Network bandwidth andlatency; sometimes topology

28

VMLayer

VnetdLayer

PhysicalLayer


Network bandwidth andlatency, sometimes topology; host load

Vnetd layer can collect all this information as a sideeffect of packet transfers

29

VMLayer

VnetdLayer

PhysicalLayer



Vnetd layer can collect all this information as a sideeffect of packet transfersand invisibly act

30

VMLayer

VnetdLayer

PhysicalLayer



Vnetd layer can collect all this information as a sideeffect of packet transfersand invisibly act•VM Migration

31

VMLayer

VnetdLayer

PhysicalLayer



Vnetd layer can collect all this information as a sideeffect of packet transfersand invisibly act•VM Migration•Topology change

32

VMLayer

VnetdLayer

PhysicalLayer



Vnetd layer can collect all this information as a sideeffect of packet transfersand invisibly act•VM Migration•Topology change•Routing change

33

VMLayer

VnetdLayer

PhysicalLayer



Vnetd layer can collect all this information as a sideeffect of packet transfersand invisibly act•VM Migration•Topology change•Routing change•Reservation

34






• Conclusions

35

VTTIF: Application Traffic Load Measurement and Topology Inference

• Parallel and distributed applications display particular communication patterns on particular topologies– Intensity of communication can also vary from

node to node or time to time. – Combined representation: Traffic Load Matrix

• VNET already sees every packet sent or received by a VM

• Can we use this information to compute a global traffic load matrix?

• Can we eliminate irrelevant communication from matrix to get at application topology?

36

Traffic Monitoring and Reduction

Host

VM

VNET

vmnet0ethz “eth0”

“Host Only” NetworkEthernet Packet Format:

SRC|DEST|TYPE|DATA (size)

VMTrafficMatrix[SRC][DEST]+=size

Each VM on the host contributes a row and column to the VM traffic matrix

Global reduction to find overall matrix, broadcast back to VNETs

Each VNET daemon has a view of the global network load

Packets observedhere

37

Denoising The Matrix

• Throw away irrelevant communication– ARPs, DNS, ssh, etc.

• Find maximum entry, a• Eliminate all entries below a

• Very simple, but seems to work very well for BSP parallel applications

• Remains to be seen how general it is

38

Offline Results: Synthetic Benchmark

39

NAS IS Benchmark

40

NAS IS Benchmark h1 h2 h3 h4 h5 h6 h7 h8

h1 19.0 19.6 19.2 19.6 18.8 13.7 19.3

h2 22.6 10.7 10.8 10.7 10.9 9.7 10.5

h3 22.2 8.78 11.2 10.4 10.1 10.5 10.5

h4 22.4 8.9 9.5 11.1 10.8 10.6 10.2

h5 22.3 10.0 9.51 9.72 11.7 10.9 11.9

h6 24.0 8.9 10.7 9.9 10.8 12.2 12.1

h7 23.2 10.0 9.7 9.5 10.3 10.2 12.0

h8 24.9 11.2 11.0 11.8 11.5 11.2 10.7 *numbers indicate MB of data transferred.

41

Online Challenges

• When to start? When to stop?– Traffic matrix may not be stationary!

• Synchronized monitoring– All must start and stop together

42

When To Start? When to Stop?

Reactive Mechanisms Proactive Mechanisms

Start when traffic rate exceeds threshold

Stop when traffic rate exceeds a second threshold

Non-uniform discrete event sampling

Provide support for queries by external agent

Keep multiple copies of the matrix, one for each resolution (1s, 2s, 4s, etc)

What is the Traffic Matrix from the last time there was at least one high rate source?

What is the Traffic Matrix for the last n seconds ?

43

Overheads (100 mbit LAN)

• Essentially zero latency impact

• 4.2 % throughput reduction versus VNET

A. Gupta, P. Dinda, Inferring the Topology and Traffic Load of Parallel Programs Running In a Virtual Machine Environment, JSSPP 2004.

44

Online: NAS IS on 4 VMs

45






• Conclusions

46

Dynamic Topology Adaptation

• VTTIF reactive mechanism run continuously

• On topology change, adjust VNET topology, adding links in priority order

• Corresponding forwarding rules also added

• Measure performance (running time) of application (BSP patterns application)

A. Sundararaj, A. Gupta, P. Dinda, Dynamic Topology Adaptation in a Virtual Network of Virtual Machines, In Submission

47

Inference and Adaptation (8 VMs, LAN)

0

10

20

30

40

50

60

70

80

90

Sec

onds

Adapt

Infer

All-to-All Bus MeshRing

48

Example Result (all-to-all, 8 VMs, LAN)

ide

alco

mp

lete

sta

r 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

0

200

400

600

800

1000

1200

1400

1600

1800

Ru

n T

ime

(S

eco

nds)

Number of Fast Path Links in Virtual Topology

No Fast Path Topology

Full all-to-all network afterstartup measurement+ reconfiguration cost

Full all-to-all frombeginning of run

Dynamic measurement andreconfiguration

49






• Conclusions

50

Adaptation With Migration

• Learn how to adapt using Virtuoso’s VM migration capabilities

• Virtuoso migration times with rsync– ~300 seconds (1.1 GB machine)– ~50 seconds (100 MB machine)

• Versioning file system approaches• Data point: CMU ISR project: 2.5-30 seconds

for personal windows VM

51

Policy Avoidance Routing

• Multi-site collaborations often stymied by interactions between per-site network security policies

• VNET opportunity: find a path on behalf of application where one exists, but is obscured

• Example: NAT Traversal– RFC 3489 / STUN (chownat)

• Example: Tunneling through initiation protocol– HTTP or SSH

52

Physical Network Measurement

• Use existing application traffic to measure underlying physical network

• Passive packet dispersion techniques– With Bruce Lowekamp, W&M

• Topology inference– With Bruce Lowekamp, W&M

M. Zangrilli and B. Lowekamp, Using Passive Traces of Application Traffic in a Network Monitoring System, HPDC 2004.

53

Integration With Resource

Prediction

Visit rps.cs.northwestern.edu for more info and downloads

54

Improving the Backbone

• Replacing the proxy star with a multisource muliticast system for higher performance and resilience

• FatNemo protocol– Arrange nodes into fat tree

S. Birrer, D. Lu, F. Bustamante, Y. Qiao, P. Dinda, FatNemo: Building a Resilient Multi-Source Multicast Fat-Tree, WCCD 2004

55

Extended Application Inference

• Offered computational load

• VM-internal performance data

• Synchronization points and waiting

• Inference of application goals

• Simple layered API for getting more application information into system

56

Making the Fast Path Fast

• Move VNET forwarder into kernel of host OS

• Guest OS device driver to directly communicate out of VM to VNET Forwarder

• Inference may make deposit message passing possible

• Goal: Minimal overhead BW and latency for using VNET, even on gigabit and faster networks

57

Leveraging Optical Networking

• Use inferred application topology to do light path setup on behalf of application

• Currently: ICAIR ODIN system, DOT network

58

Related Work• Collective / Capsule Computing (Stanford)

– VMM, Migration/caching, Hierarchical image files, Attestation• Internet Suspend/Resume (CMU/Intel)

– Your VM follows you around (will be deployed on CMU campus)• Denali (U. Washington)

– Highly scalable VMMs (1000s of VMMs per node)• CoVirt (U. Michigan)• Xenoserver (Cambridge)• SODA (Purdue)

– Virtual Server, fast deployment of services• Ensim

– Virtual Server, widely used for web site hosting– WFQ-based resource control released into open-source Linux kernel

• Virtouzzo (SWSoft)– Ensim competitor

• Available VMMs: IBM’s VM, VMWare, Virtual PC/Server, Plex/86, SIMICS, Hypervisor, DesQView/TaskView. VM/386

59

Conclusions

• Virtual machines on virtual networks as the abstraction for distributed computing

• Virtual network as a fundamental layer for measurement and adaptation

• Status– Virtuoso prototype running on our cluster– VNET 0.9 released. – VNET 1.0 (with VTTIF) in progress– Wayback versioning file system released

60

For MoreInformation

• Prescience Lab– http://plab.cs.northwestern.edu

• Virtuoso– http://virtuoso.cs.northwestern.edu

• Join our user comfort study!– http://comfort.cs.northwestern.edu

• Join our intrusion detection study!– http://ga-ids.cs.northwestern.edu

Documents

Adaptive Virtual Networking For Virtual Machine-based Distributed Computing Peter A. Dinda Prescience Lab Department of Computer Science Northwestern University