Data Plane Testing on USN, Data Plane Testing on USN, ESnet, and Internet2 NetworksESnet, and Internet2 Networks
Tom Lehman (USC/ISI)
Nasir Ghani (Tennessee Tech)
Chin Guok (ESnet)
Nagi Rao (ORNL)
John Vollbrecht (Internet2)
John Moore (MCNC)
US Dept. of Energy Office of Science
ESCC/Internet2 Joint Techs Summer Meeting
July 16, 2007
Fermi Lab
Batavia, Illinois
Hybrid MLN
2
Outline
• Dataplane testing configuration
• Dataplane test data
• Dataplane simulation
• Dataplane test conclusions
• Future Work
3
Hybrid Network Data Planes
• Basic premise of hybrid networks is the availability of both best effort routed service and deterministic dedicated resource paths, i.e., circuits
• There are many technologies available over which to construct these circuits– IP router-based Multiprotocol Label Switching (MPLS)
Label Switched Paths (LSPs) “circuits”– Ethernet VLAN based “circuits”– SONET/SDH TDM “circuits”– Wavelength Division Multiplexing (WDM) “circuits”
4
What Data Plane Technologies to Use?
• What do you want to do with your circuits?– Dedicated bandwidth connections for deterministic file
transfers?– Dedicated bandwidth & low jitter for instrument control
or interactive applications?– Connector backhaul to your IP Network?– Traffic engineering of your IP Network?
• Dynamic router-to-router circuits for traffic cut thru?
– Computer to Computer communications?• Processor to memory? block data storage system access?
– Setting up application specific topologies to create & optimize distributed application or data storage systems?
5
Data Plane Testing
• Test characteristics/performances of “circuits” constructed via different technologies; and also “end-to-end paths” constructed via concatenation of individual circuits
• Questions– What is difference between the different technologies? – How well does the concatenation/stitching work? – How well does policing/shaping work at the edge? – What happens to a flow that is policed/shaped at the
ingress edge by the time it exits the egress edge?
6
http://hybrid.east.isi.edu
Data Plane Testing
Data Plane Technology
PSC L2SC TDM LSC USN x x SDN x x x
ESNet x I2 Abilene x
I2 DCS/HOPI x x x
Netw
ork
DRAGON x x x
Data Paths Across:• ESnet• USN• Abilene• Internet2 Network• DRAGON
7
• Spirent AX4000 - Hardware based Traffic Source and Sink• External CDMA Clock allows for synchronized timestamps
Spirent AX4000 • 10 Gbps with OC192 POS / BERT / 10GbE• Two Gigabit Ethernet
Test Equipment
8
Data Collection• Approximately 75 individual tests (generally a unique path)
– tests were sourced from 1 Gbps interfaces, some from 10 Gbps– some of the measured flows had cross traffic introduced
• Tests generally included 9 measured data collection runs– 64, 500, 8000 byte MTU– 100, 500, 800 Mbps for 1 Gbps paths– 1, 5, 8 Gbps for 10 Gbps paths
• For each test run, the following data was collected:– average datarate– total packet loss– average latency– jitter profile (histogram)– transfer delay (histogram)
9
HOPI-Abilene-UltraScience Net-ESnet Test
Spirent source Washington-Force 10 Washington-Juniper T640 Chicago-Juniper T640 Chicago-Force 10 Chicago-Glimmer Glass Chicago-Force 10 Chicago-Cisco 6509 Seattle-Juniper T640 Sunnyvale-Juniper T640 Sunnyvale-Force 10 Sunnyvale-CDCI Seattle-CDCI Chicago-CDCI Chicago-Force 10 Chicago-Juniper T640 Washington-Juniper T640 Washington-Force 10 Spirent receiver
packet inter-arrival
packet end-to-end delay
Histograms
10
Circuit Description Example
• The formal description of this extended inter-network path:– Circuit type:
• usn [ethernet:tdm:ethernet]:i2dsn [ethernet:tdm:ethernet]:esnet [ethernet:pscq:ethernet]:usn[ethernet:tdm:ethernet]
– Circuit path: • usn [ORNL:CHIN]:i2dsn [CHIN:WASH]:esnet [WASH:CHIN]:usn
[CHIN:STTL:SUNV]
11
Modeling & Simulation
Overview• Discrete event simulation
• GUI interface, high re-use
• Full C/C++ interface
• Hierarchical modeling: Subnet-node-link-process
“In-House” Development• MPLS/GMPLS control: RSVP-TE, OSPF-TE, PCE
• Layer 2/3 data plane: IP/MPLS, VLAN
• Full Layer 1 support: DWDM, SONET,GFP
• Model any networks
OPNET ModelerTM Environment
12
Test Plans, Reports, and Data Repositories
• Test Report and Plans– http://hybrid.east.isi.edu DataPlane Testing
and Analysis
• Raw data repositories– http://www.csm.ornl.gov/ultranet/
SpirentMeasurements/– http://hpn.east.isi.edu/dataplane/sprint-test-
data/
13
Summary/Conclusions• All of the tested networking technologies (PSC, L2SC, TDM, LSC)
and networks (ESnet, USN, Abilene, HOPI, DRAGON) performed well both individually and when concatenated together
• There are some key differences observed between the various networking layer technologies when driven at or close to bottleneck capacity
– QoS techniques applied to router MPLS or Ethernet switched paths exhibited notably different delay behaviors versus dedicated circuit-paths (TDM)
– TDM-based infrastructures is most germane for applications requiring stringent guarantees on latency, jitter, and bandwidth protection
• Inter-layer cross-connections can be achieved in a reasonable manner by “stitching” together different network layer technologies.
– Ethernet VLANs presents the least problematic demarc (automated techniques needed to coordinate VLAN tag space)
• Future Work– impact of ingress traffic "burstiness” on end-to-end delay and loss profiles, i.e.,
both for reference and interfering cross-traffic streams.– best techniques for ingress policing and transit node QoS– vendor interoperability testing– additional network testing