Data Plane Testing on USN, Data Plane Testing on USN, ESnet, and Internet2 NetworksESnet, and Internet2 Networks

Tom Lehman (USC/ISI)

Nasir Ghani (Tennessee Tech)

Chin Guok (ESnet)

Nagi Rao (ORNL)

John Vollbrecht (Internet2)

John Moore (MCNC)

US Dept. of Energy Office of Science

ESCC/Internet2 Joint Techs Summer Meeting

July 16, 2007

Fermi Lab

Batavia, Illinois

Hybrid MLN

2

Outline

• Dataplane testing configuration

• Dataplane test data

• Dataplane simulation

• Dataplane test conclusions

• Future Work

3

Hybrid Network Data Planes

• Basic premise of hybrid networks is the availability of both best effort routed service and deterministic dedicated resource paths, i.e., circuits

• There are many technologies available over which to construct these circuits– IP router-based Multiprotocol Label Switching (MPLS)

Label Switched Paths (LSPs) “circuits”– Ethernet VLAN based “circuits”– SONET/SDH TDM “circuits”– Wavelength Division Multiplexing (WDM) “circuits”

4

What Data Plane Technologies to Use?

• What do you want to do with your circuits?– Dedicated bandwidth connections for deterministic file

transfers?– Dedicated bandwidth & low jitter for instrument control

or interactive applications?– Connector backhaul to your IP Network?– Traffic engineering of your IP Network?

• Dynamic router-to-router circuits for traffic cut thru?

– Computer to Computer communications?• Processor to memory? block data storage system access?

– Setting up application specific topologies to create & optimize distributed application or data storage systems?

5

Data Plane Testing

• Test characteristics/performances of “circuits” constructed via different technologies; and also “end-to-end paths” constructed via concatenation of individual circuits

• Questions– What is difference between the different technologies? – How well does the concatenation/stitching work? – How well does policing/shaping work at the edge? – What happens to a flow that is policed/shaped at the

ingress edge by the time it exits the egress edge?

6

http://hybrid.east.isi.edu

Data Plane Testing

Data Plane Technology

PSC L2SC TDM LSC USN x x SDN x x x

ESNet x I2 Abilene x

I2 DCS/HOPI x x x

Netw

ork

DRAGON x x x

Data Paths Across:• ESnet• USN• Abilene• Internet2 Network• DRAGON

7

• Spirent AX4000 - Hardware based Traffic Source and Sink• External CDMA Clock allows for synchronized timestamps

Spirent AX4000 • 10 Gbps with OC192 POS / BERT / 10GbE• Two Gigabit Ethernet

Test Equipment

8

Data Collection• Approximately 75 individual tests (generally a unique path)

– tests were sourced from 1 Gbps interfaces, some from 10 Gbps– some of the measured flows had cross traffic introduced

• Tests generally included 9 measured data collection runs– 64, 500, 8000 byte MTU– 100, 500, 800 Mbps for 1 Gbps paths– 1, 5, 8 Gbps for 10 Gbps paths

• For each test run, the following data was collected:– average datarate– total packet loss– average latency– jitter profile (histogram)– transfer delay (histogram)

9

HOPI-Abilene-UltraScience Net-ESnet Test

Spirent source Washington-Force 10 Washington-Juniper T640 Chicago-Juniper T640 Chicago-Force 10 Chicago-Glimmer Glass Chicago-Force 10 Chicago-Cisco 6509 Seattle-Juniper T640 Sunnyvale-Juniper T640 Sunnyvale-Force 10 Sunnyvale-CDCI Seattle-CDCI Chicago-CDCI Chicago-Force 10 Chicago-Juniper T640 Washington-Juniper T640 Washington-Force 10 Spirent receiver

packet inter-arrival

packet end-to-end delay

Histograms

10

Circuit Description Example

• The formal description of this extended inter-network path:– Circuit type:

• usn [ethernet:tdm:ethernet]:i2dsn [ethernet:tdm:ethernet]:esnet [ethernet:pscq:ethernet]:usn[ethernet:tdm:ethernet]

– Circuit path: • usn [ORNL:CHIN]:i2dsn [CHIN:WASH]:esnet [WASH:CHIN]:usn

[CHIN:STTL:SUNV]

11

Modeling & Simulation

Overview• Discrete event simulation

• GUI interface, high re-use

• Full C/C++ interface

• Hierarchical modeling: Subnet-node-link-process

“In-House” Development• MPLS/GMPLS control: RSVP-TE, OSPF-TE, PCE

• Layer 2/3 data plane: IP/MPLS, VLAN

• Full Layer 1 support: DWDM, SONET,GFP

• Model any networks

OPNET ModelerTM Environment

12

Test Plans, Reports, and Data Repositories

• Test Report and Plans– http://hybrid.east.isi.edu DataPlane Testing

and Analysis

• Raw data repositories– http://www.csm.ornl.gov/ultranet/

SpirentMeasurements/– http://hpn.east.isi.edu/dataplane/sprint-test-

data/

13

Summary/Conclusions• All of the tested networking technologies (PSC, L2SC, TDM, LSC)

and networks (ESnet, USN, Abilene, HOPI, DRAGON) performed well both individually and when concatenated together

• There are some key differences observed between the various networking layer technologies when driven at or close to bottleneck capacity

– QoS techniques applied to router MPLS or Ethernet switched paths exhibited notably different delay behaviors versus dedicated circuit-paths (TDM)

– TDM-based infrastructures is most germane for applications requiring stringent guarantees on latency, jitter, and bandwidth protection

• Inter-layer cross-connections can be achieved in a reasonable manner by “stitching” together different network layer technologies.

– Ethernet VLANs presents the least problematic demarc (automated techniques needed to coordinate VLAN tag space)

• Future Work– impact of ingress traffic "burstiness” on end-to-end delay and loss profiles, i.e.,

both for reference and interfering cross-traffic streams.– best techniques for ingress policing and transit node QoS– vendor interoperability testing– additional network testing

14

Thank-You

Questions & Comments ?

Tom Lehmantlehman@east.isi.edu