Juniper Networks QFX3500

Switch Assessment

February 2011

Juniper QFX3500 Switch Assessment P

age2

Executive Summary Juniper Networks commissioned Network Test to assess the performance of the QFX3500, a 48x10G,

4x40G Ethernet top-of-rack switch optimized for data center deployment.

Performance tests included throughput and latency measurements of unicast traffic in layer-2 and layer-

3 modes (including both cut-through and store-and-forward modes for layer-2 tests), as well as

throughput and latency of layer-2 multicast traffic.

Network Test also assessed the ability of the QFX3500 to handle storage and data traffic on a converged

network. These tests validated the switch’s ability to act as a Fibre Channel gateway while concurrently

handling Ethernet data traffic. Tests also validated support for IEEE 802.1Qbb priority-based flow control

(PFC) and determined whether concurrent storage and multicast traffic would affect switch latency.

Key results from QFX3500 testing include the following:

Sub-microsecond average latency in store-and-forward mode regardless of frame length

Line-rate throughput for all frame sizes in all test cases, both for unicast and multicast traffic

Layer-2 and Layer-3 unicast throughput figures are virtually identical

Unicast and multicast average latency are virtually identical

Support for both cut-through and store-and-forward modes of operation

IGMP snooping support for at least 1,400 multicast groups

Interoperability with the Cisco MDS 9148 Fibre Channel switch

Support for Fibre Channel over Ethernet (FCoE) and Fibre Channel gateway functionality

Support for PFC prioritization of storage traffic

No impact on latency of multicast traffic when concurrently forwarding multicast and FC/FCoE

storage traffic

Junos operating system, including all supported layer-2 and layer-3 switching and routing

protocols

The remainder of this document discusses the test results in more detail. Besides presenting the test

results, each section describes the test objective and procedure, as well as its meaning for network

architects and network managers.

Juniper QFX3500 Switch Assessment

Pag

e3

Unicast Throughput Throughput describes the highest rate at which a switch forwards traffic with zero frame loss. It’s a

critical metric, given that even a single dropped frame can have adverse effects on application

performance. This test measures throughput for unicast traffic, as defined in RFCs 1242, 2285, 2544, and

2889. Tests involved a fully meshed pattern of traffic between 48 switch ports for a duration of 60

seconds per iteration, using Spirent TestCenter as the traffic generator.

Network Test measured throughput using three switch configurations: In layer-2 mode, as a store-and-

forward device; in layer-2 mode, as a cut-through device; and in layer-3 mode, as a store-and-forward

device1. Tests were run with 64, 128, 256, 512, 1,024, 1,280, 1,518, and 9,216-byte jumbo frames.

The QFX3500 exhibited line-rate throughput for all frame sizes. In every single test case, the Juniper

switch never dropped a frame. Figure 1 summarizes throughput results for the QFX3500.

Figure 1: QFX3500 Throughput

1 As discussed in RFC 1242, a store-and-forward device caches an entire frame before making a forwarding decision, while a cut-through device begins forwarding a frame before it has been fully received. For layer-2 switching of unicast traffic, a device can be configured in either mode. For layer-3 forwarding of unicast traffic and for all multicast traffic, switches always use store-and-forward mode.

Juniper QFX3500 Switch Assessment P

age4

Unicast Latency As specified in RFC 2544, latency describes the delay introduced by the switch at the throughput rate.

Latency is arguably even more important than throughput in predicting application performance, since it

affects every frame passing through a switch, regardless of load. Some applications, such as those

involving voice and video and those used in high-frequency trading, are particularly sensitive to latency.

Network Test measured the QFX3500’s minimum, average, and maximum latency in the same three

scenarios as in the throughput tests: As a layer-2 store-and-forward device; as a layer-2 cut-through

device; and as a layer-3 store-and-forward device (with one subnet and VLAN per switch port).

When configured in layer-2 store-and-forward mode, the Juniper QFX3500 exhibited minimum unicast

latency as low as 610 nanoseconds; average unicast latency as low as 850 nanoseconds; and maximum

unicast latency not exceeding 1.1 microseconds. Moreover, store-and-forward latency did not rise

significantly as frame lengths increased. Figure 2 summarizes minimum, average, and maximum layer-2

unicast store-and-forward latency for the QFX3500.

Figure 2: Layer-2 Unicast Store-and-Forward Latency

When configured as a cut-through device, average latency for the QFX3500 ranged from 990

nanoseconds to 1.50 microseconds, and maximum latency did not exceed 1.64 microseconds.

Juniper QFX3500 Switch Assessment

Pag

e5

Sharp-eyed readers will note that cut-through latency figures increase with frame size, which is more

often a characteristic of store-and-forward devices. This is not a measurement error; as required by RFC

1242, Network Test used last-in, first-out (LIFO) measurements for store-and-forward mode and first-in,

first-out (FIFO) measurements for cut-through mode. Rather, the latency profile is explained by the

switch ASIC (application-specific integrated circuit), which caches shorter frames (in this test, those of

512 bytes and less) before making forwarding decisions, even in cut-through mode.

Figure 3 summarizes layer-2 unicast cut-through latency for the QFX3500.

Figure 3: Layer-2 Unicast Cut-Through Latency

Network Test also measured latency of the QFX3500 when configured as a layer-3 device, with different

IPv4 subnets (and layer-2 VLANs) set up on each port.

To determine layer-3 scalability, Network Test conducted the layer-3 tests twice: once with a single host

per port, and again with the Spirent TestCenter test instrument emulating 43 hosts on each port, for

more than 2,000 hosts total.

When configured in layer-3 mode, the QFX3500 delivered virtually identical latency when routing

unicast traffic between one host per port or routing among more than 2,000 hosts.

Figure 4 summarizes results from layer-3 unicast latency tests of the Juniper QFX3500.

Juniper QFX3500 Switch Assessment P

age6

Figure 4: Layer-3 Unicast Latency

Layer-2 Multicast Performance A separate set of tests focused on layer-2 multicast performance, using the aggregated multicast

throughput and latency tests described in RFC 3918. Here, the focus was on multicast performance

when the QFX3500 is configured in IGMP snooping mode. Tests used IGMPv2. One Spirent TestCenter

port acted as a multicast transmitter and the remaining 47 ports acted as multicast receivers, each

subscribed to the same 1,400 IP multicast groups.

As in the unicast tests, the Juniper QFX3500 delivered line-rate throughput for all multicast traffic,

regardless of frame length.

Tests also validated the ability of the QFX3500 to forward traffic to 1,400 IP multicast groups, with all

receiver ports subscribed to all groups.

Moreover, multicast average and maximum latency was virtually identical to unicast latency. Figure 5

compares layer-2 unicast and multicast latency.

Juniper QFX3500 Switch Assessment

Pag

e7

Figure 5: Unicast and Multicast Latency Compared

Converged Network Testing: Mixing Storage and Data Traffic Network Test examined the QFX3500’s ability to handle storage and data traffic on a converged

network. Increasingly, network architects are looking to combine legacy Fibre Channel storage traffic

with Ethernet data traffic onto converged data center networks. This convergence in turn requires data

center switches to provide Fibre Channel and FCoE functionality; to interoperate with existing Fibre

Channel equipment; to ensure lossless delivery of storage traffic; to prioritize storage traffic during

periods of congestion; and to offer low latency when combining storage and data traffic.

To assess the QFX3500’s ability to meet these requirements, Juniper and Network Test engineers

constructed a test bed connecting the QFX3500 switch over a single 8-Gbit/s Fibre Channel link with a

Cisco Systems MDS 9148 Fibre Channel switch. This interconnection was possible because up to 12

QFX3500 ports can be used for Fibre Channel connections. Engineers then configured the Spirent

TestCenter traffic generator to offer FCoE storage traffic to two QFX3500 ports, and to offer data traffic

to 40 remaining 10G Ethernet ports. In some tests, engineers deliberately attempted to overload the

FCoE ports to verify correct PFC operation. Figure 6 shows the test bed layout for converged testing.

Juniper QFX3500 Switch Assessment P

age8

Figure 6: Converged Storage/Data Test Bed

The convergence tests, which were mostly functional in nature, validated the following attributes about

the QFX3500:

The QFX3500 handles both Fibre Channel and FCoE traffic, and acts as a gateway to other Fibre

Channel devices

Using 8-Gbit/s Fibre Channel interfaces, the QFX3500 interoperates with the MDS 9148 Fibre

Channel switch from Cisco Systems

The QFX3500 supports lossless delivery of storage traffic, even during periods of congestion

The QFX3500 supports PFC, the IEEE 802.1Qbb specification

Network Test also conducted one performance test in this area to determine whether latency would be

affected when the QFX3500 concurrently handled storage and multicast data traffic. Engineers first

measured baseline latency for multicast traffic alone, using one transmitter and 39 subscriber ports,

again with 1,400 multicast groups as in previous tests. Engineers then reran the same multicast test, but

this time also offered storage traffic between two FCoE ports on the QFX3500 switch.

As shown in Figure 7, multicast latency is virtually identical in tests with and without storage traffic.

These results, along with the fact that the switch never dropped storage frames, suggest there is no

performance penalty involved when the QFX3500 handles converged data/storage traffic.

Juniper QFX3500 Switch Assessment

Pag

e9

Figure 7: The Effect of FCoE on Multicast Latency

The QFX3500 also supports other convergence features such as Fibre Channel Initialization Protocol (FIP)

snooping when the device acts as a transit switch; Enhanced Transmission Selection (ETS); and the data

center bridging (DCBX) extensions to LLDP. However, Network Test did not evaluate these features in

this round of testing.

Conclusion A key goal of these tests was to validate the performance characteristics of the Juniper QFX3500 when

used as a data center switch, especially for those data centers looking to combine storage and data

traffic on a single network. The QFX3500 delivered line-rate throughput in all tests while ensuring low

latency for both unicast and multicast traffic. Throughput was virtually identical between layer-2 and

layer-3 throughput, and between unicast and multicast latency.

The QFX3500 also proved capable in functional tests of data/storage network convergence. The switch

ensured lossless forwarding of storage traffic, with prioritization of Fibre Channel and FCoE frames

during periods of congestion. It also functioned as a gateway to other Fibre Channel equipment on the

test bed.

Finally, like other Juniper Ethernet switches, the QFX3500 supports the full range of protocols available

through Junos, the operating system for Juniper devices in the data center and beyond.

Juniper QFX3500 Switch Assessment P

age1

0

Appendix A: About Network Test Network Test is an independent third-party test lab and engineering services consultancy. Our core

competencies are performance, security, and conformance assessment of networking equipment and

live networks. Our clients include equipment manufacturers, large enterprises, service providers,

industry consortia, and trade publications.

Appendix B: Hardware and Software Releases Tested This appendix describes the software versions used on the test bed. All tests were conducted in January

2011 at Juniper’s headquarters facility in Sunnyvale, CA, USA.

Component Version

Juniper QFX3500 Junos 11.1B1.4

Cisco MDS 9148 NX-OS 5.0(1a)

Spirent TestCenter 3.60.7966

Appendix C: Disclaimer Network Test Inc. has made every attempt to ensure that all test procedures were conducted with the utmost precision and accuracy, but acknowledges that errors do occur. Network Test Inc. shall not be held liable for damages which may result for the use of information contained in this document. All trademarks mentioned in this document are property of their respective owners.

Version 2011021400. Copyright 2011 Network Test Inc. All rights reserved.

Network Test Inc. 31324 Via Colinas, Suite 113 Westlake Village, CA 91362-6761 USA +1-818-889-0011 http://networktest.com info@networktest.com