WAN in LabNSF Site Visit

John Doyle, CDS/EE/BE

Steven Low (PI), CS/EE

Harvey Newman, Physics

Demetri Psaltis, EE/CNS

Steven Yip, Cisco

March 5, 2003

netlab.caltech.edu

Reviewer concerns Narrow focus on TCP/AQM

A range of IST research at Caltech

Spanning theory, implementation, experiment, deployment

WAN in Lab a critical component

Alternatives not discussed Use spectrum of tools at different stages

How to manage and share WAN in Lab Part of Federated Emulab

Both demand and excellent support for global sharing

Experience in global collaboration, e.g. Newman’s VRVS

netlab.caltech.edu

Agenda EAS, IST Initiative, Theory program, FAST

Intellectual environment in which WAN in Lab fits (Murray, Doyle, Low)

WAN in Lab Design, capabilities, alternatives, management (Low)

Cisco example & collaboration (Yip)

Education, outreach, poster session Research talks

Projects that will use WAN in Lab

International collaboration, leverage & impact on HENP & Grids

WAN in Lab

Steven Low

netlab.CALTECH.edu

NSF Site Visit March 5, 2003

Why Testbed in IST

“A lack of wide-area testbeds would contribute to a growing tendency towards paper solutions to thesis-factory problems, leaving the real networking world short of new ideas and technologies”

“Prototypes & testbeds are required to gain acceptance of new concepts with potential user communities”

A value of testbeds is “… building and maintaining research collaborations and communities”

- NSF Workshop on Network Research Testbeds (Nov 2002)

netlab.caltech.edu

Outline Proposal summary

Basic design, equipment, costs Unique features Alternatives

Spectrum of tools Emulated delay

Community resource Demand Management software

Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid

Summary Reviewer concerns Review criteria

netlab.caltech.edu

Goal

State-of-the-art WAN High speed

2.5G 10G Large distance

50 – 200ms Controlled & repeatable experiments Reconfigurable & evolvable

1

20

1

20

fiber spool

OPM

Max path length = 10,000 kmMax one-way delay = 50ms

S

S

S

S

R

R

H

R

: server

: router

electroniccrossconnect

S

S

S

S

R

R

EDFA EDFA

500 km

netlab.caltech.edu

Equipment 26 Servers

GbE cards ( 10GbE cards) 12 routers

10 Cisco 15454 with router blades 2-port GbE, 8-channel OC48

2 Force10 E600 24-port GbE, 2-port OC48

DWDM gears 500km fiber 6 EDFA 2 Dispersion compensation modules 2 optical mux/demux

Tektronix TDS7404 Oscilloscope Integration with global network

netlab.caltech.edu

Costs 26 Servers: $104K 12 routers: $1.03M

2 Force10 E600: $280K ($340K if OC192) 10 Cisco 15454 with router blades: $750K ($810K if OC192)

DWDM gears: $148K 500km fiber: $8K 6 EDFA: $60K 2 Dispersion compensation modules: $40K 2 optical mux/demux: $40K

Tektronix TDS7404 Oscilloscope: $50K Integration with global network: $110K Personnel, software, service & maintenance

Total: $2M (NSF) + $0.67M (cost sharing)

netlab.caltech.edu

Yearly costs Year 1: $1.128K

10 servers, 5 routers, 2.5Gbps Year 2: $564K

20 servers, 8 routers, 2.5Gbps Year 3: $124K

Software development Year 4: $733K

26 servers, 10 routers, 2.5Gbps Year 5: $120K

Software development

Total: $2M (NSF) + $0.67M (cost sharing)

netlab.caltech.edu

Networking Lab

Jorgensen Lab

NetLab WAN in Lab

3 racks, 2 consoles Networking Lab

424 sq ft

Next to CACR

Easy connection to global network

Renovation (cost sharing)

New IST Building

netlab.caltech.edu

Unique capabilities WAN in Lab

Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip

Configurable & evolvable Topology, rate, delays, routing Always at cutting edge

Risky research MPLS, AQM, routing, …

Integral part of R&A networks Transition from theory, implementation,

demonstration, deployment Transition from lab to marketplace

Global resource

(a) Physical network

R1

R2

R10

1

3

20

2

1819

1

3

20

2

4

19

netlab.caltech.edu

Unique capabilities WAN in Lab

Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip

Configurable & evolvable Topology, rate, delays, routing Always at cutting edge

Risky research MPLS, AQM, routing, …

Integral part of R&A networks Transition from theory, implementation,

demonstration, deployment Transition from lab to marketplace

Global resource

R1 R2

R3R10

(b) Logical network

1 23

419

20

netlab.caltech.edu

WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip

Configurable & evolvable Topology, rate, delays, routing Always at cutting edge

Risky research Dynamic recovery, AQM, MPLS, routing, …

Integral part of R&A networks Transition from theory, implementation,

demonstration, deployment Transition from lab to marketplace

Global resource

Federated Netlab (Emulab)

Unique capabilities

Calren2/Abilene

Chicago

Amsterdam

CERN

Geneva

SURFNet

StarLight

WAN in LabCaltech

research & production networks

Multi-Gbps50-200ms delay

Experiment

netlab.caltech.edu

Outline Proposal summary

Basic design, equipment, costs Unique features Alternatives

Spectrum of tools Emulated delay

Community resource Demand Management software

Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid

Summary Reviewer concerns Review criteria

netlab.caltech.edu

Spectrum of toolslog(cost)

log(abstraction)mathsimulationemulationlive nk WANiLab

NSSSFNetQualNetJavaSim

Mathis formulaOptimizationLinear modelNonlinear modelStocahstic model

DummyNetEmuLabModelNetWAIL

HENPAbileneCalRENWAILPlanetLabCAIRNNLR

?

…we use them all

netlab.caltech.edu

Spectrum of tools

mathsimulationemulationlive nk WANiLab

Distance High High High

Speed High High Low

Realism High High Low

Traffic High Low Low

Configurable Low Medium High

Monitoring Low Medium High

Cost High Medium Low

Critical in developmente.g. Web100

netlab.caltech.edu

Emulated delay

Available technology inadequate Spirent SX/14 Link Simulator: 1ms (155Mbps) – 10s (100bps)

Adequate technology too expensive 2.5Gbps, 100ms delay: IC expert at least 2 man-years & $200K

Less realistic

1

20

1

20

S

S

S

S

R

R

S

S

S

S

R

R

High speed electronic memory

netlab.caltech.edu

HENP testbed

Ins. RTT

CWND: 5801-5815

Sylvain Ravot (Caltech/CERN)

netlab.caltech.edu

Example1: end-to-end delayY: RTT (us) Ins. RTTCWND: 5801-

5815

X:Real Time (us)

instantaneous RTT

Delay between Geneva & Chicago

average RTT

netlab.caltech.edu

Example1: end-to-end delayY: RTT (us) Ins. RTTCWND: 5801-

5815

X:Real Time (us)

instantaneous RTT

RTT=270ms12450 pkts!?

Delay between Geneva & Chicago

1500 pkt time without buildup!?

netlab.caltech.edu

Example1: end-to-end delay

X:Real Time

Y: RTT (us)

Ins. RTT

Avg RTT

Y: RTT (us) Ins. RTTCWND: 8700->4000

X:Real Time (us)

RTT=980ms!?

Passive monitoring in WANiLab can help debug

netlab.caltech.edu

Example2: 10G Expt

inst RTT

avg RTT

Losses & retransmissions

Real time s

s

Delay between Geneva & Sunnyvale

netlab.caltech.edu

Example2: 10G Expt

inst RTT

avg RTT

Retransmission without loss!?

Losses & retransmissions

Real time s

s

netlab.caltech.edu

Example2: 10G Expt

inst RTT

avg RTT

Retransmission without loss!?

Passive monitoring in WANiLab can help debug

Real time s

s

netlab.caltech.edu

Network debugging Performance problems in real network

Simulation will miss

Emulation might miss

Live network hard to debug

Enable or speed up FAST development 10GExpt: 20 people in 8 organizations for 3 months

Complete facility available only for a week

Many mysteries unresolved

WAN in Lab Passive monitoring inside network

Active debugging possible

netlab.caltech.edu

Passive monitoring

David Wei (Caltech)

Fibersplitter

DAG

RAID

TimestampHeader

GPS

Monitor

No overhead on system Can capture full info at OC48

UofWaikato’s DAG card captures at OC48 speed

Can filter if necessary Disk speed = 2.5Gbps*40/1500

= 66Mbps Monitors synchronized by GPS

or cheaper alternatives Data stored for offline

analysis

netlab.caltech.edu

Passive monitoring

David Wei (Caltech)

Fibersplitter

DAG

RAID

TimestampHeader

GPS

Monitor

Server

Server

router

router

monitor

monitor

monitor monitor

monitor

monitor

Web100, FAST monitor

netlab.caltech.edu

Outline Proposal summary

Basic design, equipment, costs Unique features Alternatives

Spectrum of tools Emulated delay

Community resource Demand Management software

Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid

Summary Reviewer concerns Review criteria

netlab.caltech.edu

DataTAG link

Ins. RTT

CWND: 5801-5815

Sylvain Ravot (Caltech/CERN)

netlab.caltech.edu

DataTAG link

Funded by EU (CERN), USA (DoE, NSF, Caltech) OC48 circuit StarLight-CERN

Upgrade to OC192 by August 2003 Linux farms

StarLight: 20 CPU (P4), 20 Syskonnect CERN: 12 CPU (P4), 12 Syskonnect

50 users, 13 institutes, 7 countries (Feb 2003) Heavy utilization

European hours: 100% reservation US hours: 25% reservation, but busy

netlab.caltech.edu

Netbed (Emulab) Funded by NSF with Cisco donations Integrates simulation, emulation, live Internet

Dummynet & VLAN Emulab Classic

University Utah: 168 PC, 5 100M Ethernet cards

Connected by 4 Cisco 6409 Testbed backplane limited to 2Gbps University of Kentucky: 48 PC, similar setup

Netbed: Federated Emulab 32 nodes in 25 sites

Heavy utilization July2002

65 user accounts (40 external) 54 projects

Feb 2003 400 user accounts 94 projects (10 Utah, 78 US, 6 Int’l)

University of Utah

netlab.caltech.edu

Management software Part of Federated Emulab

Tailor Emulab management software Jay Lepreau’s team consult on setup

Complementary to existing federated Emulab

WANiLab High speed large distance (Gbps WAN) Small network (30 nodes)

Emulab Low speed (100Mbps LAN, 10M WAN) Large network (200+ nodes)

Instantly available to Emulab community

Web accessible anywhere any time Virtual machine for network experimentation

netlab.caltech.edu

Experiment life cycle (White et al)

Experiment creation Web based sign-up form by project lead Approved by Emulab team

Experiment specification ns script or Java GUI Can download own OS, host algorithms, etc Links emulated by Dummynet nodes with specified rate,

delay, loss Experiment realization

Map target configuration to physical resources Reserve resources for each experiment Oversubscription dynamic reallocation, swap in, swap out

netlab.caltech.edu

Why Caltech: synergies Caltech networking research

FAST project: the missing experimental facility (Doyle, Low) IST Initiatives: testbed tied to rich theory program (Murray, Psaltis) Combination of theory, implementation, experiment & deployment

Synergy in research Caltech’s leadership role in IT for global HENP (Newman) Vibrant research in HENP, astronomy, geological sci, biology, visualization,

CACR Early testing ground & adopter of FAST (Newman) Availability of real data for ultrascale networks

Synergy in facility Integration with HENP networks, Abilene, CalREN XD, TeraGrid (see

Newman’s talk) Synergy with Cisco

See Yip’s talk

netlab.caltech.edu

Why Caltech: experience

Hardware Cisco’s testbed Psaltis, Yip, Hajimiri, DeHon

Software Netbed management software

Operation Newman’s group

Testbed driven by networking research IST, Theory Program, FAST, optics, scientific

computing, network coding, …

netlab.caltech.edu

Team Hardware

Yip’s team: Doraiswami (Cisco) Psaltis (EE/CNS), DeHon (CS), Hajimiri (EE) Caltech Information Tech Services, CACR

Software Lepreau’s team Low’s team: Almsberger (CS), Jin (CS), Wei (CS), Hu (CS)

Operation Newman’s team: Bunn (Physics), Ravot (Physics/CERN),

Suresh (CACR) Testbed driven by networking research

Caltech IST Institute

netlab.caltech.edu

Global research network

NLNLSURFnet

GENEVA

UKUKSuperJANET4ABILEN

E

ABILENE

ESNETESNET

CALREN

CALREN

ItItGARR-B

GEANT

NewYork

FrFrRenater

STAR-TAP

STARLIGHT

Wave

Triangle

Newman (Caltech)

WAN in LabCaltech

netlab.caltech.edu

Outline Proposal summary

Basic design, equipment, costs Unique features Alternatives

Spectrum of tools Emulated delay

Community resource Demand Management software

Why Caltech Leverage on Abilene, HENP, CalREN, TeraGrid

Summary Reviewer concerns Review criteria

netlab.caltech.edu

Reviewer concerns Narrow focus on TCP/AQM

A range of IST research at Caltech (Murray, Doyle) Spanning theory, implementation, experiment, deployment WAN in Lab a critical component

External projects in HENP, Grid & Emulab communities Alternatives not discussed

Use spectrum of tools at different stages Each complementary but not replaceable DWDM gears more realistic and cheaper

How to manage and share WAN in Lab Part of Federated Emulab Both demand and excellent support for global sharing Experience in global collaboration, e.g. VRVS

How much hardware development needed Mostly off-the-shelf (Yip) Sample system & experience from Cisco Local expertise: Psaltis (Optics), Yip (Cisc), Hajimiri (high speed IC), DeHon

(VLSI)

netlab.caltech.edu

Review Criteria Intellectual merit

Theory, implementation, experiment, deployment Must inform and influence each other intimately Approach validated by pilot project

Experimental facility tied to rich theory program

Broader impacts HENP’s global collaborations a model for future

corporations & society FAST protocols enabling technology

Shared by & stimulate external research that need high speed large distance HSTCP, Scalable TCP, TCP Westwood, AVQ, REM/PI, …

Internet as simplest complex system

netlab.caltech.edu

Review Criteria Integration of research & education

Excellent projects for undergraduates and graduates During & after development

Unique teaching platform for advanced networking, distributed systems, complex systems, optics course Bruck, Chandy, Doyle, Hickey, Low, Psaltis

Diversity 33% women grad students in Netlab 50% women postdocs and grad students in Doyle’s group

Synergy among projects Bring together 4 CISE projects (1 ITR, 1 STI, 2 pending) Leverage for additional funding and industry collaborations

netlab.caltech.edu

NSF Workshop Criteria Tested driven by research agenda

Rich and strong networking effort

“A network that can break”

Multi-user experimental facility With a clear research focus and foreseeable impact

Federated testbed Leverage on Netbed’s management software

Integrated monitoring & measurement facility Fiber splitter passive monitors

Technology transfer Strong leadership in FAST user community (Newman)

netlab.caltech.edu

Some potential projects TCP: FAST, HSTCP(Floyd, ICIR), TCP Westwood(Gerla, UCLA), Scalable

TCP(Kelly, Cambridge/CERN), XCP (Dina, MIT) AQM: REM(Low), PI(Misra/Towsley), AVQ(Srikant, UIUC) Protocol decomposition (Doyle, Low, Caltech) Network self-management (Yemini, Columbia) Content distribution (Bruck, Low, Caltech, Xu, Washington U) Optical switching (Low, Psaltis, Caltech) Network separation theory (Doyle, Low, Caltech Paganini, UCLA) Real-time control over high performance networks (Dolye, Low, Murray,

Caltech) Simple Optics Smart Router (SOSR) (Yates, AT&T Research) Optical protection, recovery (Yates, AT&T Research; Nirmalathas,

Melbourne U) Dynamic lightpath configuration & provisioning (Tucker, Melbourne U) Active probing (Veitch, CUBIN) Passive monitoring (Veitch, CUBIN) Building & testing firewalls (Hoffman, U of Victoria) High performance active network node (Turner, Washington)