In VINI VeritasRealistic and Controlled Network Experimentation

Andy Bavier Nick Feamster* Mark Huang

Larry Peterson Jennifer Rexford

Princeton University *Georgia Tech

How to Validate an Idea?

Fixed, shared among many experiments Runs real routing software Exposes realistic network conditions Gives control over network events Carries traffic on behalf of real users

Simulation

Emulation

Small-scaleexperiment

Livedeployment

VINI

Scientific Value

The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ (I found it!) but ‘That’s funny …’ -- Isaac Asimov

Move off the emulator, into the wild Opportunity for more ‘that’s funny’ moments

Avoid “Fallacy of Misplaced Concreteness” Simulation and emulation are important tools Modeling abstracts general properties from reality

Philosophy: the devil may be in the details… But insights and soundness are found there too

“Controlled Realism” Start with a controlled

experiment Relax constraints,

study effects Result: an operational

virtual network that’s Feasible Valuable Robust Scalable, etc.

Topology

Actual network

Arbitrary, emulated

Traffic

Real clients, servers

Synthetic or traces

Network Events

Observed in operational network

Inject faults, anomalies

Overview VINI requirements

Fixed, shared infrastructure Flexible network topology Expose/inject network events External connectivity and routing adjacencies

Strategy for building VINI PL-VINI: prototype on PlanetLab Experimental results Timeline

Fixed Infrastructure

Deploying VINI nodes in National LambdaRail, Abilene with Gigabit links

Shared Infrastructure

Experiments given illusion of dedicated h/w

Flexible Topology

VINI supports arbitrary virtual topologies

Network Events

VINI exposes, can inject network failures

External Connectivity

s

c

Experiments can carry traffic for real end-users

External Routing Adjacencies

s

c

BGP

BGP

BGP

BGP

Experiments can participate in Internet routing

PlanetLab VINI

Build VINI from PlanetLab, a global testbed for distributed servicesBegun in 2002700 nodes at 336 sites in 35 countries 600 projects and 2500 researchersServes 3-4 TB/day to ~1M clients

MyPLC: PlanetLab software distributionAnyone can run their own private PlanetLab

PlanetLab Experiments

Simultaneous experiments in separate VMs Each has “root” in its own VM, can customize

Reserve CPU, network capacity per experiment

Virtual Machine Monitor (VMM)(Linux++)

NodeMgr

LocalAdmin

VM1 VM2 VMn…PlanetLab node

PL-VINI: Prototype on PlanetLab

Feasible? prototype on public PlanetLab Enable experiment: Internet In A Slice

XORP open-source routing protocol suite (NSDI ’05) Click modular router (TOCS ’00, SOSP ’99)

Clarify issues that a VINI must address Unmodified routing software on a virtual topology Forwarding packets at line speed Illusion of dedicated hardware Injection of faults and other events

XORP: Control Plane

Goal: real routing protocols on virtual network topologies

BGP, OSPF, RIP, PIM-SM, IGMP/MLD

XORP can run in a PlanetLab VM

XORP(routing protocols)

PlanetLab VM

User-Mode Linux: Environment

Interface ≈ network PlanetLab limitation:

Experiments cannot create new interfaces

Run routing software in UML environment

Create virtual network interfaces in UML

XORP(routing protocols)

UML

eth1 eth3eth2eth0

PlanetLab VM

Click: Data Plane

Performance Avoid UML overhead Move to kernel, FPGA

Interfaces tunnels Click UDP tunnels

correspond to UML network interfaces

Filters “Fail a link” by blocking

packets at tunnel

XORP(routing protocols)

UML

eth1 eth3eth2eth0

Click

PacketForwardEngine

Control

DataUmlSwitch

element

Tunnel table

Filters

PlanetLab VM

Resource Isolation

Issue: Forwarding packets in user space PlanetLab sees heavy use CPU load affects virtual network performance

Property Depends On Solution

Throughput CPU% received PlanetLab provides CPU reservations

Latency CPU scheduling delay

PL-VINI: boost priority of packet forward process

Intra-domain Route Changess

c

1176

587 846

260

700

6391295

2095

902

548

233

1893

366

Watch OSPF route convergence on Abilene

856

Ping During Link Failure

70

80

90

100

110

120

0 10 20 30 40 50

Pin

g R

TT

(m

s)

Seconds

Link down Link up

Routes converging

Abilene RTT: 73ms

TCP Throughput

0

2

4

6

8

10

12

0 10 20 30 40 50

Meg

abyt

es t

rans

ferr

ed

Seconds

Packet receiv ed

Zoom in

Link down Link up

Arriving TCP Packets

2.1

2.15

2.2

2.25

2.3

2.35

2.4

2.45

17.5 18 18.5 19 19.5 20

Meg

abyt

es in

str

eam

Seconds

Packet receiv ed

Slow start

Retransmitlost packet

PL-VINI enables a user-space virtual networkto behave like a real network on PlanetLab

Attracting Real Users

Could have run experiments on Emulab Goal: Operate our own virtual network

Carrying traffic for actual usersWe can tinker with routing protocols

We expect that:PlanetLab services will subscribe to VINI

network architectures to access Gb/s Experiments will advertise routes via BGP

Timeline

PL-VINI• PlanetLab• Resource resv• CPU priority

You arehere

NLR-VINIAbilene-VINIJapan-VINI• PCs• VINI OS• MyVINI• Xen• Exchange traffic with ISPs

2007

NLR-VINIAbilene-VINI• PCs• PlanetLab OS• MyPLC• Gigabit layer 2• eBGP uplinks to friendly ISPs

Fall 2006 2008

NLR-VINIAbilene-VINIJapan-VINI???-VINI• Other GREN• PC + FPGAs, NPs• Create layer 2 “on the fly”

Other features?

The End

Questions?

The End

URL: http://www.vini-veritas.net Questions?

Backup slides

Conclusion

VINI = evolution of PlanetLab Installing VINI nodes in NLR, Abilene Download and run Internet In A Slice MyPLC MyVINI as code diverges

Build, run, modify your own VINIWe expect there to be many VINIs

http://www.vini-veritas.net

Timeline

Conclude with a timeline instead? Like the one for Gibson.

Experiments on the top, infrastructure on the bottom, “You are here.”

Today: IIAS, PL-VINI Next: RCP, VINI-NLR What other experiments?

Ongoing Work

Improving realismExposing network failures and changes in the

underlying topologyParticipating in routing with neighboring

networks Improving control

Better isolationExperiment specification

Performance is bad

User-space Click: ~200Mb/s forwarding Can do a lot with 200Mb/s

20 experiments can have dedicated 10Mb/s nationwide networks

Improving performance is ongoing workAllow experiments to load custom Click

modules into the VINI kernel

PL-VINI SummaryFlexible Network Topology

Virtual point-to-point connectivity Tunnels in Click

Unique interfaces per experiment Virtual network devices in UML

Exposure of topology changes Upcalls of layer-3 alarms

Flexible Routing and Forwarding

Per-node forwarding table Separate Click per virtual node

Per-node routing process Separate XORP per virtual node

Connectivity to External Hosts

End-hosts can direct traffic through VINI Connect to OpenVPN server

Return traffic flows through VINI NAT in Click on egress node

Support for Simultaneous Experiments

Isolation between experiments PlanetLab VMs and network isolation

CPU reservations and priorities

Distinct external routing adjacencies BGP multiplexer for external sessions

PL-VINI / IIAS Router

XORP: control plane UML: environment

Virtual interfaces

Click: data plane Performance

Avoid UML overhead Move to kernel, FPGA

Interfaces tunnels “Fail a link”

XORP(routing protocols)

UML

eth1 eth3eth2eth0

Click

PacketForwardEngine

Control

DataUmlSwitch

element

Tunnel table

What’s New with VINI?

Integration of routing w/Internet Better isolation Real topologies Inject events

“Controlled Realism” Control:

Reproduce results Methodically change or

relax constraints

Realism: Long-running services

attract real “customers” Forward high traffic

volumes (Gb/s) Robustly handle

unexpected events

Topology

Actual network

Arbitrary, emulated

Traffic

Real clients, servers

Synthetic or traces

Network Events

Observed in operational network

Inject faults, anomalies