Request for feedback on proposed direction for CHEETAH

1

Request for feedback on proposed direction for

CHEETAH

Outline A quick status report on CHEETAH

Proposed direction discussion: What's our goal for the CHEETAH network:

eScience network or a scalable GP network?

Bandwidth sharing modes: Book-Ahead (BA) or Immediate-Request (IR)?

If IR, what applications?

Core or edge network?

Malathi VeeraraghavanUniversity of Virginia

Sept. 22, 2006

2

CHEETAH project status

Network deployment data plane control plane

Networking software Network service Applications

eScience general-purpose

Basic research on circuit/VC networking

3

CHEETAH network - data plane linksGbEthernet and SONET

TN PoP

Controlcard

GbE/10GbEcard

GA PoP

Controlcard

GbE/10GbEcard

SN16000

Controlcard

OC192card

OC-192

GbE/10GbEcard

End hosts

NC PoP

SN16000 SN16000

GaTech

End hosts

End hosts

ORNL

OC192cards

NCSUOC192card

OC-192 (via NLR/SLR/NCREN)

via NCREN

UVa

CUNY Via Nysernet/HOPI Via Vortex/HOPI

GbEGbE

GbEs

GbE

GbE

GbEGbE

GbE

4

CHEETAH network - control plane links

TN

Controlcard

GbE/10GbEcard

GA

Controlcard

GbE/10GbEcard

SN16000

Controlcard

OC192card

GbE/10GbEcard

End hosts

NC

SN16000 SN16000

GaTech

End hosts

End hosts

ORNL

OC192card

NCSUOC192card

UVa

CUNY

Internet2

ns5

ns5ns5

IPsec device

OpenswanIPsec softwareon Linux end hosts

Design goal: scalable GMPLS network

Call setupmessages

5

Networking software Sycamore switch comes with built-in GMPLS

control-plane protocols: RSVP-TE and OSPF-TE

We developed CHEETAH software for Linux end hosts: circuit-requestor

allows users and applications to issue RSVP-TE call setup and release messages asking for dedicated circuits to remote end hosts

CircuitTCP (CTCP) code

Described in two ICC06 papers and a JSAC submission

6

Network service

On-demand circuit-switched service for 1Gb/s dedicated host-to-host circuits

Call setup delay: 1.5sec Sycamore gave us a proprietary build for hybrid

GbE-SONET-GbE circuits No standard yet for such hybrid circuits Sets up 7 OC3s and VCATs them to carry a GbE

signal In contrast, their GMPLS standards

implementation for pure-SONET circuits incurs a call setup delay of 166ms (2-hop)

7

Applications

eScience: Terascale Supernova Initiative File transfers: demo'ed but

held up by Cray X1 network I/O problems Ensight remote visualization: demo'ed but

connectivity between NCSU CHEETAH-drop site and physicist's office is low speed

Conclusion: other factors, not the network! general-purpose:

web caching video apps

8

Basic research on bandwidth-sharing mechanisms in circuit/VC networks

Immediate-Request (IR) mode Purpose: to determine what type of applications are

well served by this mode Key finding: if m, the link capacity expressed in

channels, is 10, IR (Immediate Request) mode of bandwidth sharing leads to poor util. or high blocking

Published in an IEEE Globecom 2006 paper Book-ahead bandwidth-sharing mechanisms

Developed a novel discrete-time Markov chain model for book-ahead bandwidth-sharing mechanisms

Results are submitted to IEEE/ACM Trans. on Networking)

9

Outline check

A quick status report on current CHEETAH Proposed direction discussion:

What's our goal for the CHEETAH network: eScience network or a scalable GP network?

Bandwidth sharing mode: Book-Ahead (BA) or Immediate-Request (IR)?

If IR, what applications? Core or edge network?

10

Observation

"Many e-science experiments ... are optimized to provide maximum throughput to a few facilities, as opposed to moderate throughput to millions of users, which is the raison d'etre for commercial networks."

11

eScience networks

eScience network requirements Number of users small Hard to achieve high utilization; also not impt. Overprovision network to keep call blocking

rate low Focus on creating software to allow scientists

to automatically provision high-speed application-specific topologies: AST, UCLP, OSCARS, USN scheduler, BRUW

Bandwidth-sharing algorithms of less concern

12

General-purpose commercial networks

Have to be scalable: large number of users Metcalfe's statement: Value of a network

increases exponentially with the number of users

High utilization is an important goal Low call blocking probability or low

waiting time for resources Focus on efficient bandwidth-sharing

algorithms

13

Circuit/VC service on GP commercial networks

Just for ISPs/enterprise admins: needs similar to eScience

router-to-router circuits limited number of users high-bandwidth, long-held circuits low price not a high priority need BA mode of bandwidth sharing

For end users large number of users can only offer moderate BW and limited call holding

times IR mode of sharing becomes feasible

14

Outline check





15

BW sharing modes in circuit/VC networks

Mean waiting time is proportional to mean call holding time Can afford to have a queueing based solution when m is

small if calls are short

Large m Moderate throughput Small m

Short calls Long callsBank teller Doctor's office

High throughput

immediate-requestwith call blocking + retries("call queueing")(video, gaming)

immediate-requestwith delayed-start times("call queueing") (file transfers)

book-ahead

m is the link capacityexpressed in channelse.g., if 1Gbps circuitsare assigned on a 10Gbps link,m = 10

16

Impact of increasing m at different values of link utilization Ud

100

101

102

103

0

0.2

0.4

0.6

0.8

1

Ud=90%

Ud=80%

Ud=60%

Ud=40%

m

PQ

100

101

102

1030

200

400

600

800

1000

100

101

102

1030

200

400

600

800

1000

100

101

102

1030

200

400

600

800

1000

100

101

102

1030

200

400

600

800

1000

Ud=90%

Ud=80%

Ud=60%

Ud=40%

m=10

Pq=41%

Pro

b.

of a

rriv

ing

job

findi

ngal

l m c

ircui

ts b

usy

Off

ered

load

: ca

ll ar

rival

rat

e/ca

ll d

epar

ture

rat

e

Low-rate per-call circuits High-rate per-call circuits

Link capacity expressed in channels

17

Impact of mean call holding time /1

0 5 10 15 20 25 3010

0

101

102

103

104

105

1/ (minutes)

N

0 5 10 15 20 25 300

6

12

18

24

30

0 5 10 15 20 25 300

6

12

18

24

30

0 5 10 15 20 25 300

6

12

18

24

30

0 5 10 15 20 25 300

6

12

18

24

30

0 5 10 15 20 25 300

6

12

18

24

30

0 5 10 15 20 25 300

6

12

18

24

30

E[W

d] (m

inute

s)

m=1000,=1call/hour

m=100,=1call/hour

m=10,=1call/hour

m=10,=10calls/hour

m=10

m=100m=1000

Num

ber

of

port

s ag

gre

gatin

g tr

affic

on t

o th

e li

nk

Mea

n w

aiti

ng

time

for

dela

yed

calls

: per host call-generation rate Ud: 90% )1(

1][

dd Um

WE

'

18

Main findings of analysis Two key parameters:

If m is small (per-circuit BW is high) and mean call holding time is large

then need BA to avoid long waiting times

and mean call holding is small (file transfers) then use "call queueing"

If m is large, switch hardware costs increase N, number of aggregation ports, high level of demultiplexing high

19

Relate BW sharing modes to network types

Bandwidth-sharing mechanisms

Book-Ahead (BA) Immediate-Request (IR)

eScience networks

very large file transfers need high-BW and long holding time + remote viz. need to reserve other resources such as displays

None?

general-purpose networks

circuit service to only ISPs/enterprise admins - router-to-router circuits

circuit service for end users- host-to-host + router-to-

router (end-to-end)- partial-path router-to-

router circuits on congested links (called in by end user)

20

Support for the BA mechanism of bandwidth sharing

Since RSVP-TE does not have parameters for BA calls (call duration, start time), this mode is not implemented in switch controllers

Cannot utilize the BW management software implemented in switch controllers as part of GMPLS control-plane software

Need an external scheduler to manage bandwidth into the future

Easiest to make it centralized - one per domain The BA mode is necessary for high-BW, long-

held calls

21

BA implementation approach Develop and "standardize" protocols for scheduler-to-

scheduler signaling for interdomain circuits (one centralized scheduler per domain) e.g., Chin Guok (ESNET)'s WSDL spec. for ESNET-Abilene

testing (OSCARS-BRUW) Implement scheduler and test with other networks Create software tools to enable scientists and

ISP/enterprise admins to visualize network topologies and request appropriate circuits/VCs

High-BW, long-held: Therefore AAA is a must Path being pursued by DRAGON, USN, OSCARS, UCLP

22

Argument: IR is just a "now" in BA

Difficult to have BA and IR coexist without some form of bandwidth partitioning BA allows for high-BW, long-duration calls IR calls will suffer a high call blocking rate if

supported through BA scheduler (the "add-now-as-an-option-in-scheduler" solution)

Should you admit an IR call if it arrives a few seconds before start time of a BA call and hope it completes before the BA call start time, or reject the call and waste bandwidth?

If BW is partitioned, then implement scalable solution for IR - distributed bandwidth-management

23

Support for the IR mechanism of bandwidth sharing

Switches have built-in (G)MPLS control-plane software (RSVP-TE/OSPF-TE)

Bandwidth management is part of RSVP-TE switch controller software Hence it is distributed bandwidth

management Need to limit call holding time -

reminders for renewals and automatic release

Moderate-to-high per-call bandwidth

24

Is an opportunity being missed if distributed IR bandwidth sharing mode is not explored?

What opportunity? Four reasons:1. Enable the creation of large-scale circuit/VC

networks with moderate-rate circuits that can support a brand new class of applications

economic value for the networking industry

2. A "reservations-oriented" mode of networking to complement today's connectionless Internet

ala airlines that complement roadways

3. Could prove useful to FIND, GENI, net-neutrality4. Alternative pricing models for bandwidth

25

Outline check





26

What "brand new class of applications?"

Large-m (moderate BW) Video, video, video Gaming Remote software access

Small-m (high BW) short-held calls Async storage Web and P2P file transfers

27

Video applications

Improve quality of conferencing, telephony, surveillance, entertainment and distance-learning by a significant degree

Expend bandwidth for a higher-quality, lower latency, multi-camera, auto-movement, auto-mixing experience

Make the "flat world" flatter Energy savings/environmental benefits Moderate bandwidth - IR with call

blocking/retries

28

Gaming applications

Current gamers buy personal graphics cards Players talk of "lag" caused by differences in

graphics processing speeds Moderate-speed circuits can enable a new class of

games in which rapidly-changing scenes are possible compare movies in which multiple story lines

keep scenes changing vs. gaming scenes Players connect to graphics servers Data transferred is not GL commands, but rather

rendered data (doable?) Moderate bandwidth - IR with call blocking/retries

29

Remote software access

Remote software access Reduce computer administration cost Personal computers vs. machine rooms I loaded 22 new applications on my new laptop

Instead: connect and run! Virtual Computing Laboratory: Mladen Vouk, NCSU

Moderate bandwidth - IR with call blocking/retries

30

Asynchronous storage

Asynchronous storage depots will lower costs for backups disaster recovery

Need for increased storage grows with multimedia files

High bandwidth, short calls IR with delayed start

31

Larger files in web sites and P2P apps

Multimedia files in web sites Increase the use of video/audio files in all sorts of web

sites instead of ASCII My own course PPT files: I use audio sparingly because of

bandwidth Think assembly instructions for electric fans, furniture

Kinesthetic learning - show me a video Think hotel web pages

Show me exactly where the beach is relative to my room; do I have a balcony - saying it in text format is one thing; seeing it in a video format quite another!

Content distribution network, mirroring & web caching

High bandwidth, short calls IR with delayed start

32

Do these apps really require circuit/VC networks with IR mode or can they simply use higher-BW IP based networks?

See analogous transportation networks (reason 2) reservations-oriented airlines network reservationless roadways network

Hypothesis: The socialistic mode of bandwidth sharing on the Internet discourages individual investment in network bandwidth (reason 4) should we pay for bandwidth with tax dollars - "free"

for the whole community? "Tragedy of the commons" (Tanenbaum)

should we create a network where individuals can pay for bandwidth on congested links more directly? - think higher-toll HOV lanes

33

Invest in a complementary reservations-oriented network

Build a scalable circuit/VC network in which bandwidth is shared in IR mode Scalability will create "Metcalfe's value" Provides an opportunity to finally recoup our

investment in (G)MPLS technologies standards creation effort implementation: Cisco, Juniper, Sycamore, Movaz

Assign at least a few of the optical testbeds that we are investing in now to study whether this IR mode of bandwidth sharing can help with our understanding of net-neutrality, economic growth, FIND questions

IR more natural in data world unlike in airlines (BA)

34

Outline check





35

Core vs. edge networks Should the GMPLS IR scalable network be developed

first as a core network or as a set of edge networks that feed into an IP-based core network? Edge-network oriented apps: HOV bypass

Video Gaming Remote software access

Core-network oriented apps: "3TCP connections" with wide-area TCP connection through GMPLS network Asynchronous storage Large web sites: CDN/web caching

36

HOV bypass

The "end-to-end" in the CHEETAH name We chose Ethernet in LANs and SONET in

WANs for the data-plane, given the dominance of these two technologies, with the thinking that this would make "end-to-end" circuits affordable

BUT, even adding control-plane software modules to each Ethernet switch in a departmental closet, enterprise closet, etc. is quite expensive

37

HOV bypass

Further observation if a link is lightly loaded, we don't really need to

reserve bandwidth for a particular flow Combining these two:

Concept of "partial-path circuits" Determine if there is a bottleneck link on the

end-to-end IP path If so, send signaling request to a router on the

bottleneck link asking for a BW reservation on bottleneck link (just as with HOV lanes)

38

Key idea

Use control-plane signaling request from end user or application to ask IP router for a "bypass" reservation and to get ready to isolate packets of its flow (Policy Based Routing) Unlike ATM cut-through VCs solutions that required

IP routers to somehow determine which flows to handle via ATM VCs Ipsilon: long-lived flow detection

Comparable to the user who calls ahead and reserves a seat for a flight, before showing up at the airport

39

Use IR mode for HOV bypass

Enterprise network

WAN-access router

Edge network(e.g., NYSERnet)

Enterprise network

WAN-access router

Enterprise network

WAN-access router

Enterprise network

WAN-access router

Backbone network (e.g., Abilene)

Edge network(e.g., NCREN)

bypasscontroller

dynamicallysetup MPLStunnel forperiod of flow

access: likely congested links

likely lightly loaded links

NYC PoPNC PoP

40

Edge network(e.g., NYSERnet)

Using CHEETAH ideas in enterprise access for the HOV bypass MSPPs were being deployed in enterprises to aggregate

voice and IP traffic on to SONET WAN access links

Enterprise network

WAN-access router bypass

controller

add additionalOCxx circuit(router needsextra or higher-BWinterface already plugged in)

But increasingly Metro Ethernet is replacing SONET MPLS seems more suitable

41

Core network solution:3TCP connections

CAG: CHEETAH Application GatewayA web cache (using squid) with integrated cheetah software

Roadways

RoadwaysAirlines

Airport Airport

Appears that a comparable reason exists here for taking a flight: long-distance travel

42

Splitting high-BDP TCP connection

Improves end-to-end delay even if all three TCP connections used IP paths Micah Beck, Logistical networking Martin Swany, Phoebus Sigcomm 05 Purdue poster Many others

43

Cheetah improvement

Have the wide-area path go through a circuit/VC network if RTT is 50ms, even if circuit rate is 10Mbps when

Internet path bottleneck link rate is 100Mbps, there is a crossover file size beyond which the circuit path is faster - even with BIC

Trigger for call setup through CHEETAH arrives from web proxy connections initiated by clients Thus, users not physically connected to CHEETAH still

use CHEETAH service CDN and mirror apps will be triggered by web

server-to-local mirror writes

44

CHEETAH-HOPI interconnection

McLean, VA

10GbE

CHEETAH

HOPI network: courtesy of Rick Summerhill

NC

GATN

Webcache

Webcache

Webcache

Webcache

Webcache

Webcache

Webcache

Webcache

VLAN

MPLS

SONET

45

CHEETAH as a core network

Job of a core network is to connect edge networks

Application just described only connects web servers, web caches, storage devices to CHEETAH SONET switches at PoPs

Could also provide router-to-router IR-triggered circuits between PoPs Allow admin-triggered setups Network management software for load

monitoring and automatic triggers

46

Add routers and servers/storage to CHEETAH PoP

Webcaches

Web servers/mirrors

Storagedevices

......

GA PoP

Webcaches

Web servers/mirrors

Storagedevices

......

NC PoP

Webcaches

Web servers/mirrors

Storagedevices

......

TN PoPRouters

Routers

Routers

OC192

OC192

47

Conclusions

Proposed direction: What's our goal for the CHEETAH network:

eScience network or a scalable GP network?


If IR, what applications? Long-distance file transfers - web caching/CDN/mirrors

and storage + router-to-router triggers

Core or edge network?

Interesting research: delayed-start BW-sharing

Documents

Request for feedback on proposed direction for CHEETAH