Download pdf - 10GE Testing AppNote

Spirent TestCenter Application Note #1

10-Gigabit Ethernet TestingApril 2006

P/N 71-000689 REV A

Spirent Communications, Inc.26750 Agoura Road Calabasas, CA 91302 USA

Support ContactsU.S. E-mail: [email protected] Web: http://support.spirentcom.com Toll Free: +1 800-SPIRENT (+1 800-774-7368) Phone: +1 818-676-2616 Fax: +1 818-880-9154 China E-mail: [email protected] Web: http://support.spirentcom.com.cn Toll Free: +1 800-810-9529 (mainland China only) Phone: +86 10 8233 0033 (rest of the world) Fax: +86 10 8233 0022 Europe E-mail: [email protected] Web: http://support.spirentcom.com Phone: +33 (0) 1 61 37 22 70 Fax: +33 (0) 1 61 37 22 51

Copyright 2006 Spirent Communications, Inc. All Rights Reserved. All of the company names and/or brand names and/or product names referred to in this document, in particular, the name Spirent and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners. The information contained in this document is subject to change without notice and does not represent a commitment on the part of Spirent Communications. The information in this document is believed to be accurate and reliable, however, Spirent Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in the document.

Limited WarrantySpirent Communications, Inc. (Spirent) warrants that its Products will conform to the description on the face of order, that it will convey good title thereto, and that the Product will be delivered free from any lawful security interest or other lien or encumbrance. Spirent further warrants to Customer that hardware which it supplies and the tangible media on which it supplies software will be free from significant defects in materials and workmanship for a period of twelve (12) months, except as otherwise noted, from the date of delivery (the Hardware Warranty Period), under normal use and conditions. To the extent the Product is or contains software (Software), Spirent also warrants that, if properly used by Customer in accordance with the Software License Agreement, the Software which it supplies will operate in material conformity with the specifications supplied by Spirent for such Software for a period of ninety (90) days from the date of delivery (the Software Warranty Period). The Product Warranty Period shall mean the Hardware Warranty Period or the Software Warranty Period, as applicable. Spirent does not warrant that the functions contained in the Software will meet a specific requirement or that the operation will be uninterrupted or error free. Spirent shall have no warranty obligations whatsoever with respect to any Software which has been modified in any manner by Customer or any third party. Defective Products and Software under warranty shall be, at Spirent's discretion, repaired or replaced or a credit issued to Customer's account for an amount equal to the price paid for such Product provided that: (a) such Product is returned to Spirent after first obtaining a return authorization number and shipping instructions, freight prepaid, to Spirent's location in the United States; (b) Customer provides a written explanation of the defect or Software failure claimed by Customer; and (c) the claimed defect actually exists and was not caused by neglect, accident, misuse, improper installation, improper repair, fire, flood, lightning, power surges, earthquake, or alteration. Spirent will ship repaired Products to Customer, freight prepaid, within ten (10) working days after the receipt of defective Products. Except as otherwise stated, any claim on account of defective materials or for any other cause whatsoever will conclusively be deemed waived by Customer unless written notice thereof is given to Spirent within the Warranty Period. Spirent reserves the right to change the warranty and service policy set forth above at any time, after reasonable notice and without liability to Customer. TO THE EXTENT PERMITTED BY APPLICABLE LAW, ALL IMPLIED WARRANTIES, INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT AND FITNESS FOR A PARTICULAR PURPOSE, ARE HEREBY EXCLUDED, AND THE LIABILITY OF SPIRENT, IF ANY, FOR DAMAGE RELATING TO ANY ALLEGEDLY DEFECTIVE PRODUCT SHALL BE LIMITED TO THE ACTUAL PRICE PAID BY THE CUSTOMER FOR SUCH PRODUCT. THE PROVISIONS SET FORTH ABOVE STATE SPIRENT'S ENTIRE RESPONSIBILITY AND CUSTOMER'S SOLE AND EXCLUSIVE REMEDY WITH RESPECT TO ANY BREACH OF ANY WARRANTY.

10-Gigabit Ethernet TestingThe development of high throughput and low latency 10-Gigabit network switches and routers is challenging. Network equipment manufacturers of switches and routers with 10-Gigabit Ethernet network interfaces are challenged to provide switch fabrics that have high availability 10 Gigabits-per-second line rate forwarding that meets the requirements of converged, multiservice networks that demand low latency performance. This application note introduces Spirent Communications new test tool platform, Spirent TestCenter, how it addresses the critical aspects of 10-Gigabit Ethernet performance testing, and the network protocols that affect 10-Gigabit Ethernet layer 2 and layer 3 switch performance. Spirent TestCenter is uniquely suited to assist network equipment manufacturers to meet the challenges of successfully delivering 10GbE network devices.

In this document... Overview of 10-Gigabit Ethernet Network Devices . . . . 4 10-Gigabit Ethernet Network Interfaces for Enterprise Networks . . . . 5 10-Gigabit Ethernet Network Two Rates in One . . . . 6 New Applications and Protocols for 10-Gigabit Ethernet . . . . 8 Scalable 10GbE Networks Create Test Challenges . . . . 12 10-Gigabit Ethernet Test Considerations 10GbE is not Gigabit Ethernet . . . . 12 Traditional Switch Testing and 10GbE Considerations . . . . 13 Test Setup for 10GbE using the RFCs . . . . 14 Line Rate Testing for 10-Gigabit Ethernet Switches and Routers . . . . 14 Key Considerations for 10GbE Tests . . . . 17 Conclusion . . . . 18 Spirent TestCenter Products for 10GbE Switch and Router Testing . . . . 19

Spirent Communications Application Note

| 3

10-Gigabit Ethernet Testing Overview of 10-Gigabit Ethernet Network Devices

Overview of 10-Gigabit Ethernet Network DevicesThe deployment of 10-Gigabit Ethernet (10GbE) ports in network switching and routing equipment is increasing rapidly. Most major network equipment manufacturers (NEMs) offer layer 2 switches, layer 3 aware switches, and layer 3 routers. These network devices have various combinations of copper 10/100/1000Mbps and fiber Gigabit Ethernet (GE) combined with 10-Gigabit Ethernet (10GbE) ports. 10GbE is becoming a prevalent interface on network servers, firewalls, and appliances. Typically, the switch configurations that are equipped with 10GbE network interfaces are:

Fixed, managed switches these switches have a fixed number of copper or fiber GE ports with two or four 10GbEports. Thye cannot be expanded with additional ports. They can be stacked (interlinked) to form a larger network. Modular switches/ routers these are switches with a chassis that have individual blades for GE and 10GbE. The blades are removable, and the user can expand or change the port type and rate mix as required. These switches have the ability to connect directly to GE and 10GbE servers, 10GbE storage area networks, network attached storage, network appliances such as firewalls, and intrusion detection/ protection devices running GE links. Data center switches These are fixed multiple port, data center switches, some with up to 24 ports of 10GbE that are used in machine room applications in main distribution facilities in large networks that aggregate multiple 10GbE links into a single switch for distribution to different devices for multiple applications. These switches are used to run high performance applications in parallel. Network appliances Today, there are 10GbE intrusion detection devices, and upperlayer application switches that provide multiple-gigabit throughput over 10GbE.

There is a large variety of switch and router products that offer between 12 and 48 ports of Gigabit-rate capable ports combined with two or four ports of 10GbE. These switches are used in workgroup and stackable switch applications in small to medium sized Local Area Networks (LAN) that may have up to 2,000 thousand users (host clients) on the network. In larger LANs, multiple stacks of interlinked switches are connected with a 10GbE backbone or trunk line. In some networks, 10GbE backbones are used to interconnect two or more campus networks within 40 kilometers of each other, and even up to 80 kilometers with special 10GbE optical transport interfaces. These are the kinds of LAN switches that most of us are familiar with and have seen as new product introductions since early in 2003.

4

| Spirent Communications Application Note

10-Gigabit Ethernet Testing 10-Gigabit Ethernet Network Interfaces for Enterprise Networks

10-Gigabit Ethernet Network Interfaces for Enterprise NetworksThe target applications for Enterprise network equipment that deploys 10GbE optical and copper network interfaces are determined by the IEEE803.2ae and IEEE802.3ak standards. Typical applications of 10GbE interfaces are shown in Table 1. Included in Table 1 are the fundamental interface specifications. Many of these interface specifications are found in product literature for network equipment and are relevant to where the link is deployed. The 10-Gigabit Ethernet story does not end in the LAN with the LAN layer 2 protocol. The IEEE802.3ae standard has defined a Wide Area Network protocol.Table 1. Network Interface Specifications for 10gbe in Enterprise Network Applications Driven by the IEEE802.32ae Standard

10GbE Network Interfaces

Enterprise Network Applications

Layer 2 10GbE Protocol

Nominal Line Rate & Clock Tolerance10.3125 Gb/s, 100ppm

Standard Laser Optical Wavelengths

Type of Fiber, or Copper Cable62.5m MMF 50m MMF

Maximum Reach (Meters)

10GBASE-SR

Data Center, wiring closet, machine room, stackable switches Campus or Metro

Serial LAN

850nm, Short wavelength serial 310nm, Long wavelength serial 1550nm, Extra long wavelength serial 4 different wavelengths, long wave laser

2 to 33m 2 to 300m

10GBASE-LR

Serial LAN

10.3125 Gb/s, 100ppm

10m SMF

2 to 10km

10GBASE-ER

Metro

Serial LAN

10.3125 Gb/s, 100ppm

10m SMF

2 to 40km

10GBASELX4

Campus or Data Center

LAN

3.125 Gb/s per lane, 100ppm

62.5 m MMF 50 m MMF 50 m MMF 10 m SMF

2 to 300m 2 to 240m 2 to 300m 2 to 10,000m


| 5

10-Gigabit Ethernet Testing 10-Gigabit Ethernet Network Two Rates in One

Table 1.

Network Interface Specifications for 10gbe in Enterprise Network Applications Driven by the IEEE802.32ae Standard (continued)


Enterprise Network Applications


Nominal Line Rate & Clock Tolerance3.125 Gb/s per lane, 100ppm


Type of Fiber, or Copper CableTwinaxial, shielded, crossover (IEC 610763-113 plug standard)


10GBASECX4

Data Center, wiring closet, machine room, stackable switches

LAN

ae

15m

10-Gigabit Ethernet Network Two Rates in One10-Gigabit Ethernet supports two layer 2 protocols, a LAN and a WAN. The most widely implemented protocol is commonly called 10GbE LAN. It runs at a 10.3125Gbps line rate. In terms of the IEEE802.3ae standard, it is called 10GBASE-R. The LAN protocol is used in LANs, wiring closets, machine rooms, and campus and enterprise network applications. The LAN protocol may be implemented from a CE router interface to a service provider edge router interface (see Table 1 on page 5). A second layer 2 protocol for 10GbE is defined and is commonly known as 10GbE WAN. In 2005, 10GbE WAN made its appearance in switches and routers. In IEEE802.3 terminology, this protocol is called 10GBASE-W. It is supported by a frame encapsulation mechanism, defined by the IEEE802.3ae in Clause 50, called the WAN Interface Sublayer (WIS). The WIS mechanism is the 10GBASE-W definition that has been adapted from the ANSI T1.416-1999 (SONET STS-192c/SDH VC-4-64c) physical layer specifications. 10GBASE-W specifies three network interfaces: 10GBASE-SW, 10GBASE-LW, and 10GBASE-EW (see Table 2 on page 7).

6


10-Gigabit Ethernet Testing 10-Gigabit Ethernet Network Two Rates in One

Table 2.

Network Interface Specifications for 10GbE in Metropolitan Network Applications Driven by the IEEE802.32ae Standard


Metropolitan Network Applications


Nominal Line Rate & Clock Tolerance


Type Of Fiber, or Copper Cable


10GBASE-SW

Data Center, wiring closet, machine room trunks to routers Campus or Metro

Serial WAN

9.95328Gbps, 100ppm

850nm, Short wavelength serial 1310nm, Long wavelength serial 1550nm, Extra long wavelength serial

62.5m MMF 50m MMF

2 to 33m 2 to 300m

10GBASE-LW

Serial WAN

9.95328Gbps, 100ppm

10m SMF

2 to 10km

10GBASE-EW

Metro

Serial WAN

9.95328Gbps, 100ppm

10m SMF

2 to 40km

There are some significant differences between 10GBASE-R and 10GBASE-W. The 10GBASE-W protocol operates at a 9.95328Gbps line rate that conforms to the requirements of SONET STS-192c and SDH VC-4-64c frame rates. It matches the 9.58464 Gb/s payload rate of SONET. The IEEE802.3ae standard states, The purpose of the WIS is to allow 10GBASE-W equipment to generate Ethernet data streams that may be mapped directly to STS-192c or VC-4-64c streams at the PHY level, without requiring MAC or higher-layer processing. This means that its design is meant for a direct PHY-toPHY interface, a point-to-point connection. A 10GBSAE-W port should be connected to another 10GBASE-W port. The 10GBASE-W implements the minimum framing, scrambling, and error detection for SONET. It implements the sufficient Path, Section and Line overhead fields that are required to maintain a SONET link. 10GBASE-W emulates a SONET link using SONET encapsulation of the Ethernet frame. On the network, it looks like a SONET frame, but it does not have the jitter, synchronous operation, electrical interface, or various performance and line controls that a standard fully SONET compliant link would possess. 10GBASE-W is not directly interoperable with OC-192c SONET/SDH; one cannot attach a 10GBASE-W interface to an OC-192c SONET/SDH interface. The IEEE8023.ae standard requires that one 10GBASE-W interface be connected to another 10GBASE-W interface.


| 7

10-Gigabit Ethernet Testing New Applications and Protocols for 10-Gigabit Ethernet

An example of this is the connection of two routers that have 10GbE Ethernet and OC192c SONET interfaces. When the two routers are connected by two 10GBASE-SW links, the routers switching fabric can easily move the SONET encapsulated Ethernet frames seamlessly to the OC-192c interface that is connected to a core optical network. Rates, framing, and error detection are aligned between the two different PHY. Within a service provider network 10GbE is used in LAN or WAN modes depending on the requirement. The WAN protocol can be used to link to a core network at 10Gb/s rates. The WAN protocol encapsulates the Ethernet frame in a fixed SONET header to allow SONET devices to forward it over OC-192c SONET/SDH links. The operator of the router, from a maintenance viewpoint, would see two SONET links, not an Ethernet link and a SONET link. This is intended to make links easier to maintain in a SONET environment. In metropolitan network applications this translates into the ability to attach Ethernet switches and routers to core network devices that are attached to the SONET cloud. A service provider point-of-presence can connect a 10GBASE-W link to a carriers central office network equipment that is using SONET/SDH or optical transport equipment. Now and here is the catch the whole scheme of the IEEE802.3as standard is to enable Ethernet to participate in true end-to-end applications that makes use of compatible infrastructure in the LAN and allow Ethernet to be readily adapted to MAN environments with facility to interconnect with a network device that is attached to the optical core network. This is made possible because the underlying rate at the MAC layer is 10Gb/s, the MAC layer rate is constant and is common to both 10GbE LAN and WAN protocols. The WIS layer adapts the MACs 10Gb/s rate to be compatible with SONET/SDH. By doing this, a 10GBASE-W link inherits the lower cost and ease of maintenance inherent in 10GASE-R links. Due to the constant rate MAC, a switch or router fabric can handle switching 10GBASE-R to 10GBASE-W across the fabric in the same box. This flavor of 10GbE, 10GBASE-W, adds new complexities to performance testing of switches and routers, depending upon where the equipment is deployed in the network.

New Applications and Protocols for 10-Gigabit EthernetThere are several areas of networking and internetworking that drive the growth of 10GbE. Fundamentally, a number of new, IP-based network protocols, and time-sensitive network applications, are increasing the need for more network bandwidth. Data center applications add to the need for higher data throughput and low latency switching because switches are increasingly requested to interconnect with GE and 10GbE servers and network appliances that have multi-gigabit data rate throughput. Today, every type of network device for machine room and data center application has 10GbE interfaces available for deployment. The switch must handle these rates with layer 2 and/or layer 3 traffic without adding performance degradation at the higher network layers. IT infrastructure managers expect that layer 2 switching is never to be a part of their problem. The implementation of several bandwidth management and provisioning protocols that affect GE and 10GbE performance are shown in Table 3 on page 9. The measurement of

8



throughput, forwarding rates, and latency with these various protocols, turned on or off, with various combinations, is critical to the design of the network. There are more GE and 10GbE links present in the network than ever before, so it is important to know that network devices can handle planned traffic loads under segmented and prioritized conditions. The switch must maintain the quality of the users experience on the network.Table 3. Layer 2 and Layer 3 Protocols that Affect Forwarding Performance in Switches OSI Layer Affected Layer 2 (MAC layer) Effect on the Switch or Router Increase the length of packet headers by 4 bytes for each packet. At high 10Gb/s speeds switches must forward larger packets with very low latency. VLAN control field data must be inspected for each packet and this increases packet-processing time. Multiple VLANs per port force switches to be tested for throughput per VLAN, VLAN leakage (mis-forwarding), and over subscription that can flood ports In Ethernet end-to-end applications over the WAN it requires Layer 2 to Layer 3 mapping to maintain CoS. Switches spend more time tracking and forwarding. IEEE 802.1D Quality of Service management, MAC bridges, support for multicast networks in multimedia applications (grouped MAC addresses) Layer 2 (MAC layer) More segregation of bandwidth and network domains over individual LANs, that cause more packet inspection, increase in forwarding table sizes, and buffer memory management. This increases potential for higher latency. Switch must maintain throughput in the presence of misordered and lost frames per VLAN. Switch must inspect VLAN priorities and maintain throughput (forwarding rate) over all separate VLANs

Protocols IEEE 802.1Q, Virtual LAN (VLAN)


| 9


Table 3.

Layer 2 and Layer 3 Protocols that Affect Forwarding Performance in Switches (continued) OSI Layer Affected Layer 2 (MAC layer) Effect on the Switch or Router Traffic on a single port or across multiple ports must support different customer LANs on the service provider network. Links can be aggregates with up to 8 GE links aggregating to 1 10GbE link, and 10GbE links to 40Gbps links. Forwarding performance and maintenance of QoS over segregated LANs combined with the dedication of bandwidth and network domains per customer, because more packet inspection, increased packet lengths due to keys and tags and increased complexity of forwarding tables, and buffer memory management. This has potential for higher latency, incorrect forwarding, and VLAN leakage must be measured. Both Port and per stream QoS and forwarding rates should be measured on aggregated links. Other measurements include port and aggregated stream failover.

Protocols IEEE802.3-Clause 43 Link Aggregation and IEEE 802.1ad Bridged Local Area Networks, Virtual Bridged Local Area Networks,

Internet Protocol version 6 (IPv6) 40-byte header length Extension headers Dual Stack operations IPv6 tunneling

Layer 3 (IP Layer)

Longer standard packet header that supports header extensions. This requires additional packet inspection of a larger packet header with more fields. IPv6 and IPv4 protocol must be supported in a dual-stack mode on a per port basis. This requires additional packet inspection, which increases the possibility for higher latency and stresses memory buffers.

RFC 2474 Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers

Layer 3 (IP Layer)

Uses the DS field of the IP header to mark packets with values (called code points) that prioritize one type of traffic over another based on class and cost. It is used in QoS, CoS and layer 2 to Layer 3 mapping applications (MPLS). This, although not a guaranteed service, requires switches to perform additional header inspection and can cost processing time and increase latency.

10 |



Table 3.

Layer 2 and Layer 3 Protocols that Affect Forwarding Performance in Switches (continued) OSI Layer Affected Layer 2 and Layer 3 Effect on the Switch or Router Service provider networks have two types of routers that interact. Label Edge Routers (LER) at the provider edge and Label Switching Routers (LSR) within the provider core. MPLS provides a mechanism for IP flows from many LAN customers to enter a large service provider IP/ MPLS network and to separately maintain QoS for each customer over the backbone links between the routers in the provider network. LERs must inspect and keep track of many different network domains, and perform deep and complex, packet inspection. This increases processing time and forwarding latency. LSRs label switch the packets and forward them. The LSRs must demonstrate low Layer 2 latency. Time sensitive applications like IP telephony and video are affected by MPLS services.

Protocols Multi-Protocol Label Switching (MPLS) MPLS VPN VPLS

IP Multicast

Layer 2 and Layer 3

A one-to-many protocol used for switching and routing of audio, video, and conferencing of web casts. Switches must keep track of grouped MAC addresses and VLANs, where there can be many users per group and thousands of groups per port. The switch must be able to allow users to join and leave these groups without affecting QoS and throughput. Poor latency performance may disrupt audio and video transmissions. Switch fabrics must be able to handle IPv4, IPv6, VLANs, Unicast, and Multicast across the entire fabric.

Routing Protocols

Layer 3 (IP Layer)

Switches must be able to maintain Layer 2 throughput, forwarding rate and low latency in the presence of routing protocol overhead (control plane traffic). A single GE and 10GbE router may be aware of thousands of networks, with several hundred users per network that equates to forwarding over a few million flows.

Spirent Communications Application Note | 11

10-Gigabit Ethernet Testing Scalable 10GbE Networks Create Test Challenges

Scalable 10GbE Networks Create Test ChallengesToday, there is an ever-growing increase in the number of GE and 10GbE ports in a single blade. The challenge for NEMs is that they must test and measure the performance of these high capacity, high port count devices with all of the protocols listed in Table 3 on page 9. The simple layer 2 and layer 3 test over the data plane that measure throughput, packet loss, and latency may not be sufficient in this scalable 10GbE environment. The selection of the right test tool is critical. The test tool must be able to scale, while tracking multiple protocol details within every port, over large numbers of ports. As outlined in Table 3 on page 9, being able to test switches over their ports, with multiple VLANs per port, and with different QoS priorities over different bandwidth provisioning levels per port, requires the ability to discretely track the test traffic. When a failure occurs, it is insufficient to only know the port the errored traffic came from. It is much better and faster to find the source of errors, if the test traffic is tracked on a per stream basis, as opposed to only a per port basis. If the test Engineer knows that error traffic was sourced from the exact port, with the exact individual stream, with the exact VLAN, and the exact QoS setting for that stream, then the chances of isolating the source of the error increases greatly. This saves the test engineer a considerable time in debugging problems. The test tools built by Spirent Communications are designed with 10GbE scalability testing in mind. Spirents new test tool platform, Spirent TestCenter, provides the ability to track up to 24 different parameters on a per stream basis in real time with real-time feedback while running a test. Every stream is tracked, on every port, in a scalable manner. For bandwidth provisioning applications, every streams transmit rate, and every streams receive rate on every stream, on every port can be determined with Spirent TestCenter in a single test. Up to 32,767 clients can be tracked with stream traffic with real time measurement on a single port in the Spirent TestCenter test tool; it is the most scalable test tool in the test and measurement industry.

10-Gigabit Ethernet Test Considerations 10GbE is not Gigabit EthernetThe approach to testing 10GbE is similar to the testing of GE. However, there are several important differences between 10GbE and GE. The traditional switch test strategies for GE are not comprehensive enough for 10GbE. When testing 10GbE, the differences between 10GbE and GE must be taken into account. Further, since 10GbE is readily configured as a LAN or WAN interface, this aspect must be considered in the test plan strategy. Another important consideration is interframe gap. In GE, the interframe gap (IFG) is a constant value; it is a single byte length value. Once the IFG is set to 12 bytes for a given packet size, that translates the packets transmit speed into a line rate transmission at 1Gb/ s. This is not the case for 10GbE. The IFG at 10GbE is derived over a predefined range of IFG lengths; it presents an average IFG, and therefore an average line rate. The result is that the IFG ranges from as small as 9 bytes to as large as 15 bytes while maintaining an average 12-byte IFG. This creates problems for network devices with their packet handling mechanisms. This key difference from GE must be considered as part of the 10GbE test plan.

12 |


10-Gigabit Ethernet Testing Traditional Switch Testing and 10GbE Considerations

The test tools built by Spirent Communications are designed with 10GbE testing in mind. Spirent TestCenter has capabilities and features that are critical to the testing of 10GbE. Spirent TestCenter does account for the differences in GE and 10GbE testing. The tool accommodates the nuances in testing the 10GbE interface. This section discussed what is required in a test tool to accurately test 10GbE and its common interaction with GE. The sum of these considerations is that: (1) 10GbE requires additional scalability testing, (2) 10GbE testing requires higher resolution test tools with improved accuracy, and (3) the test tool must be flexible enough to allow the test engineer to discretely test and measure the different implementations of 10GbE (i.e 10GBASE-R and 10GBASE-W interfaces).

Traditional Switch Testing and 10GbE ConsiderationsThe tests defined in RFC 2889 and RFC 2544 are applicable, in principal, for all Ethernet speeds and network interface types from 10Mbps to 10Gbps. RFC 2544 and RFC 2889 will be referenced as the RFCs for efficiency, or as RFC test when general points are applied. The basis for the following discussion is centered over the traditional Ethernet switch tests Requests For Comment (RFC) written by the IETF:

Layer 2 data plane tests: RFC 2889 Benchmarking Methodology for LAN Switching Devices with its associated RFC 2285 Benchmarking Terminology for LAN Switching Devices. Layer 3 data and control plane tests: RFC 2544 Benchmarking Methodology for Network Interconnect Devices with its associated RFC 1242 Benchmarking Terminology for Network Interconnection Devices.

These four RFCs were written between 1991 and 2000. While all of the tests defined in the RFCs are critically important to quantify the performance of network switches, major new functionality has been added to both layer 2 and layer 3 network devices since their standardization. Table 3 on page 9 points to the most significant of these new feature sets, industry protocols, and bridging standards. The IEEE802.3ae standard was ratified in 2002, written well after the RFC tests. The RFC tests have no concept of 10GbE. They do a nice job of covering 10Mbps through 1Gbps. The major point here is to run the mandatory RFC switch tests on 10GbE and on GE with the considerations described in the following section. As discussed on page 12, 10GbE does not behave on the network like GE. So, while the RFC switch tests are applicable, in principal, to 10GbE, there are serious practical considerations that must be added to the test plan for 10GbE.


10-Gigabit Ethernet Testing Test Setup for 10GbE using the RFCs

Test Setup for 10GbE using the RFCsA few fundamentals on 10GbE are noteworthy for 10GbE RFC test setup. These considerations point out how 10GbE differs from the test RFCs, and the requirements for 10GbE.

The speed of 10-Gigabit Ethernet is 10,000,000,000 bits/sec (10Gb/s, 10Gbps) at the medium access layer (MAC). A bit time is 1/10,000,000,000 seconds. The interframe gap is 9.6 nanoseconds for 10-Gigabit Ethernet. This is still 96-bit times. 10GbE is a full-duplex protocol. There is no half-duplex mode in 10GbE. 10GbE supports two protocols: LAN and WAN. Only one protocol can be used in comparative tests between two DUTs. This is due to different data rates per protocol. 10GbE WAN has a different frame encapsulation scheme and line rate than 10GbE LAN. 10GbE WAN and 10GbE LAN must be tested separately on the switch. One cannot test one of the protocols and assume the other protocol will work. Frame sizes that pass RFC performance test specifications with 10GbE LAN may fail on 10GbE WAN and vice versa. The RFCs specify a 30-second test duration per frame size iteration. This is often not long enough to stress packet buffers on switches and routers. Longer test durations of up to 300 seconds are quite common. Test for the larger more scalable 10GbE devices may be run for multiple hours, or even days, with expectations of no packet loss. While these tests are not per any RFC, they are being conducted. Special considerations are required for these tests. At 10GbE rates, there are carrier class switches that forward 10Gb/s traffic to 40Gb/s interfaces. To do this, they have an extremely large packet buffer memory. Buffer depth presents an issue if the test does not run long enough to fill up the buffer. A special technique that monitors frame loss in real time may be required. The test tool should offer this capability to test deep packet buffer memory. One technique to reduce test time is to report packet loss in real time while the test is running, as opposed to reporting man hours later at the end of a batch mode test.

Line Rate Testing for 10-Gigabit Ethernet Switches and RoutersIn line rate tests with 10-Gigabit Ethernet, there is a critical consideration that affects the setup of the traditional RFC test and any other special performance test. The decision on how to set up the performance test must take into account the IFG mechanism for 10GbE that was previously mentioned. A brief explanation is provided for clarity. Recall the earlier discussion in which it was pointed out that 10GbE does not have a constant IFG byte length. The IFG varies between 12 and 15 bytes to produce an average of 12 bytes at line rate. The variation of the IFG is beyond the scope of this document.

14 |


10-Gigabit Ethernet Testing Line Rate Testing for 10-Gigabit Ethernet Switches and Routers

The net result is that the actual IFG, which affects the final transmission rate on the line, will be slower than the theoretical 10GbE line rate. Certain packet sizes are affected by the average IFG generating mechanism in 10GbE. A 10GbE interface will transmit between 12 and 15 bytes of interfame gap at a given frame size. What this means is that in some frame sizes, the theoretical 10GbE line rate will not be achieved. So, how does one perform a 10GbE line rate test on a switch port with so many different frame sizes that almost ensure that line rate will be never be achieved? To test at the 10GbE line rate, the test tool must have Deficit Idle Count (DIC) compensation. The test tool must allow the test to be run with DIC enabled or disabled. Fundamentally, DIC adds or subtracts up to 3 bytes to or from the nominal average 12byte interframe gap in order to maintain the 10GbE frame rate. The range of IFGs transmitted by the 10GbE interface can be from 9 to 15 bytes, but the average is 12 bytes. In order to achieve the maximum 10GbE throughput, the test tools and devices under test must have DIC enabled. The pattern in Table 4 on page 15, will faithfully repeat. There will be less difference in frames per second rates between DIC enabled and DIC disabled as the frames increase in size. If the switch is not tested with DIC enabled, and the device is placed into a DIC-enabled network, the probability increases that the DIC-enabled network interface will overrun the receive frame buffer of the device with DIC compensation disabled. This problem occurs with smaller frame lengths. DIC provides a higher gradient for frames per second rates, so in the event of packet loss, a more accurate line rate can be measured and reported with DIC enabled.Table 4. Transmit Rate Differences at the 10Gb/s Line Rate with Deficit Idle Count

Frame Size (bytes)64 65 66 67 68 69 70 71 72

Frames per second rate (DIC enabled)14,880,952 14,705,882 14,534,883 14,367,816 14,204,545 14,044,943 13,888,888 13,736,263 13,586,956

Frames per second rate (DIC disabled)14,880,952 14,204,545 14,204,545 14,204,545 14,204,545 13,586,956 13,586,956 13,586,956 13,586,956

Difference in Frame Rates0 501,337 330,338 163,271 0 457,987 301,932 149,307 0


10-Gigabit Ethernet Testing Line Rate Testing for 10-Gigabit Ethernet Switches and Routers

The first recommendation that comes from this is to initially test the switch with DIC enabled. This will stress the switch far more than with DIC disabled by subjecting it to many varying frame rates. The second recommendation is to run through a considerable range and number of small frame sizes, for both odd and even frame sizes. Throughput, packet loss, and forwarding rates must be evaluated until one is satisfied that the switch is forwarding at the 10GbE line rate over many frame sizes. The third recommendation is to make sure the range of packet sizes that are test is wide. The switch should be tested in small frame size steps (increments) from 64 bytes to 16Kb frame lengths. In Table 5 on page 16, the standard RFC frame sizes and frames per second rates are shown with DIC enabled and disabled with a 12-byte IFG. Notice that with frame sizes that are modulo 4, that no impact from DIC is seen until the 1518 frame size is encountered (reference blue highlighted table cells). This may appear like a subtle anomaly of the RFC test, or 10GbE, but it is not. When all possible frames sizes are considered and the differences in frame rates between DIC enabled and DIC disabled (Table 4) is considered there will be many frame sizes that will pass line rates tests with DIC disabled but may fail line rate tests with DIC enabled. Spirent recommends that throughput, frame loss and forwarding rate tests be conducted with DIC enabled.Table 5. 10-Gigabit Ethernet Line Rates With Standard RFC 2544 and 2889 Frame Sizes

Frame Size (bytes)64 128 256 512 7681 1024 1280 1518

Frames / second rates (DIC enabled)14,880,952 8,445,945 4,528,985 2,349,624 1,586,294 1,197,318 961,538 812,743

Frames / second rates (DIC disabled)14,880,952 8,445,945 4,528,985 2,349,624 1,586,294 1,197,318 961,538 811,688

Interframe Gap (100% line rate)12 bytes 12 bytes 12 bytes 12 bytes 12 bytes 12 bytes 12 bytes 12 bytes

1 The 768 frame size is not specified as MUST in the RFCs.

For example, if a routine run of the standard RFC frame sizes performed over a 10GbE interface without DIC, the DUT may pass 100%. Trouble may arise if that same port interconnects with another port with DIC enabled running off-standard frame sizes. The DUT may have been tested at less than line rate for non modulo 4 frame sizes. It is easy to be caught off ones guard, even in a well known test like RFC 2544 or 2889. As shown in Table 3 on page 9, the newer network protocols create frame sizes that are not part of RFC 2889 and 2544, as they were originally written. Table 5 on page 16, selects a few fundamental frame sizes that exceed the RFC test requirements. These frame

16 |


10-Gigabit Ethernet Testing Key Considerations for 10GbE Tests

sizes should be tested across the entire switch fabric with their header fields populated with realistic forwarding values seen in virtual and bridged networks. Todays switches must be tested beyond the traditional RFC, using protocols with headers that segment traffic in support of QoS as well as in converged network applications. Table 6 does not list all of the possible influences on critical Ethernet frame sizes that stress switch performance with the many protocols listed in Table 3 on page 9. The point illustrated is this: simply running a vanilla-flavored RFC 2889 and RFC 2544 switch test and claiming that ones device is ready for the world of 10GbE is not advisable.Table 6. 10-Gigabit Ethernet Frame Rates with Non-standard RFC Frame Sizes

Frame Size(s) (bytes)

Frames per second (100% line rate)961,538 822,368 812,743 808,538 138,243 135,339

Interframe Gap

Application driver

1280 1500 1522 1526 9022 9216

12 bytes 12 bytes 12 bytes 12 bytes 12 bytes 12 bytes

IPv6 minimum MTU Ethernet MTU VLAN tag VLAN tag and MPLS Label Jumbo frame Popular jumbo frame

Key Considerations for 10GbE TestsThe items listed here reinforce previous points and provide cautions for testing the performance of 10GbE switches and routers.

For line rate tests Deficit Idle Count (DIC) should be enabled unless the DUT/SUT cannot support it, or an agreement is in place to test without it. The switch or router should be performance tested with DIC enabled and with DIC disabled. This applies to the 10GbE LAN and WAN protocols Most test equipment uses statistical transmit and receive event counters. These often display results in real time (e.g. real time event counters). At the high data rates of 10GbE, real-time statistical event counters may not be capable of displaying what is actually being transmitted or received on the line. They will report rates other than the theoretical line rate for that frame size. The test equipment should have special transmit and receive counters. These special counters are required to accurately reflect the transmit and receive rates that are determined in the actual test and reflected in the test report. Note that, at the slower speed of 1Gbps, sampling mechanisms that produce real-time statistical event counters can keep pace with the data rate. The problem is usually not seen at 1Gbps.


10-Gigabit Ethernet Testing Conclusion

Latency measurements are dramatically different between 10GbE LAN and 10GbE WAN. The 10GbE WAN will have a larger average latency and range compared to 10GbE LAN. Average, minimum and maximum latency for both 10GbE WAN and 10GbE LAN should be measured. For latency measurements, the test equipment should have 10-nanosecond measurement resolution and an accuracy rating of 40 nanoseconds. 10GbE supports several types of optical and copper transceivers. Latency test results may vary according to the technology of the transceivers that are used in the test. A low quality transceiver can add latency to the performance results.

ConclusionWhat 10GbE deployment means for NEMS, service providers, carriers and their customers, is that new test methodology is required to test switches with 10GbE interfaces in a comprehensive manner. The RFC tests should be run, as well as tests that include Virtual LANs (VLANs), IPv6 addressing and tunneling, IP multicasting, and Quality of Service. Another influence that stresses switch fabrics is subnets per VLAN for IPv4 and IPv6. The summary point is that switches and routers are being asked to simultaneously perform more tasks on a single port, with more ports, at higher speeds, at lower cost. The primary challenges for switch and router vendors are:

Deeper frame and packet header inspections due to more active controls in frame headers High speed, tagging, mapping, and tracking of many more header field data to network segments Perform the functions in bullets 1 and 2 for thousands of clients on a single port, across multiple ports, over an entire switch fabric To maintain large memory buffers for forwarding tables. This is intermixed between physical and virtual networks. Switches and routers with 10GbE network interfaces must be able to forward layer 2 traffic at line rate at all packet sizes, not just the standard packet sizes, and with jumbo frames.

Table 7 on page 19 and Table 8 on page 19 provide product information for your reference. For more details on Switch Test Methodologies, please refer to Spirents Test Methodology Journals for RFC 2544 and RFC 2889.

18 |


10-Gigabit Ethernet Testing Spirent TestCenter Products for 10GbE Switch and Router Testing

Spirent TestCenter Products for 10GbE Switch and Router TestingThe Spirent TestCenter products that support 10GbE testing are shown in Table 7 and Table 8 on page 19.Table 7. 10-Gigabit Ethernet Test Hardware Modules and Related Transceivers

Model

Description

Application

10GbE Test Technologies with LAN and WAN Protocol SupportXFP transceivers: 850, 1310nm, and 1550nm XFP transceivers: 850, 1310nm, and 1550nm XENPAK LAN transceivers: 850nm, 1310nm, and 1550nm XENPAK LAN/WAN transceiver: 1310nm CX-4 Copper, LAN transceiver

XFP-1001A

Standard scalability and performance, 1-port, 10GbE test module High scalability and performance, 1-port, 10GbE test module Standard scalability and performance, 2-port Multi-MSA test module

Traffic generator and statistical measurement analyzer Traffic generator and statistical measurement analyzer High port density traffic generator and statistical measurement analyzer

XFP-2001A

MSA-1001A

XFP-1001A

XFP-2001A

MSA-2001A



Table 8.

Spirent TestCenter Software Packages for 10-Gigabit Ethernet Testing

Model

Description

Application

10GbE Test Technologies with LAN and WAN Protocol SupportIPv4, IPv6, layer 2 and layer 3 traffic generation, QoS, Diffserv, error injection, custom packet generator, capture, statistics and real-time measurements/charts Deficit Idle Count

BPK-1001A

Packet generator and analyzer base package A

Traffic generator and statistical measurement analyzer

TPK-1000

RFC-2544 with VLAN network device benchmark test package

Layer 3 RFC switch and router testing and beyond with IPv6, VLAN support Layer 2 RFC switch and router testing and beyond with IPv6, VLAN support Key metropolitan and enterprise protocols Key metropolitan and enterprise protocols Multicast routing

TPK-1001

RFC-2889 with VLAN switching benchmark test package

Deficit Idle Count

BPK-1002A

STP/RSTP/PVST+ base package A

STP/RSTP/PVST+ and perVLAN Spanning Tree Protocol state machines Multiple Spanning Tree Protocol Multicast registration IGMPv1/ v2/v3 and MLDv1/v2 Deficit Idle Count

BPK-1014A BPK-1003A

Multiple Spanning Tree base package A IGMP/MLD host IP multicast base package A

BPK-1004A

Unicast routing base package A

Unicast routing

IPv4 and IPv6 interior and exterior gateway routing protocols: RIPv1/v2, RIPng, OSPFv2/v3, IS-IS, IS-ISv6, BGP-4 and BGP+ Deficit Idle Count

BPK-1005A

Multicast routing base package A

Multicast routing

Multicast registration IGMPv1/ v2/v3 and MLDv1/v2 Deficit Idle Count

20 |



Table 8.

Spirent TestCenter Software Packages for 10-Gigabit Ethernet Testing (continued)

Model

Description

Application

10GbE Test Technologies with LAN and WAN Protocol SupportRFC 2547bis Layer 3 VPNs, Martini-draft Layer 2 VPNs (PWE emulation), Virtual Private LAN Service - LDP (VPLS LDP), Virtual Private LAN Service - BGP (VPLS LDP), Layer 3 IPv6 VPNs Deficit Idle Count

BPK-1006A

MPLS/LDP/RSVP-TE base package A

MPLS support
