ADVA FSP 150 ProVMe - eantc.de

ADVA FSP 150 ProVMePerformance and Functionality Test Report

Introduction

EANTC was commissioned by Intel under the Intel®Network Builders program to perform independenttests of the ADVA FSP 150 ProVMe (F2.6) solution.

ADVA’s FSP 150 ProVMe combines an open platformfor hosting of virtual network functions with multilayerbusiness service demarcation in a single networkelement. We were engaged to verify ADVA’s claimsregarding integration of fiber-based business accesssolutions with open VNF-hosting platforms for NetworkFunction Virtualization (NFV) of fiber-based businessaccess solutions and open-source virtual infrastructuremanagement (VIM) systems for Network Function Virtu-alization (NFV). ADVA claims that FSP 150 ProVMecan be configured with OpenFlow, OpenStack andNetconf/YANG-supporting Software DefinedNetworks (SDN) and NFV-centric networks, creatingsignificant advantages over legacy proprietary solu-tions.

In our series of tests we reviewed FSP 150 ProVMe’sperformance and functionality. EANTC confirmed thatFSP 150 ProVMe can replace legacy customer prem-ises routers with virtualized, integrated solutions,without sacrificing any of the functionality or perfor-mance.

Specifically, we covered three key capabilities:

• The VNF lifecycle management process wasdemonstrated with standard OpenStack Junorelease showing the ability to openly integrate intotypical NFVI architectures.

• The performance improvement promises of SingleRoot I/O Virtualization (SR-IOV) versus OpenvSwitch (OVS) were evaluated and confirmed witha range of tests evaluating the forwardingperformance.

• FSP 150 ProVMe can provide a wide range ofhardware-based support functions improvingperformance without consuming computeresources. Our test cases verified Access ControlLists (ACLs), port mirroring, network clocksynchronization and VNF-performance assurancefunctions running independently from the server.

As our evaluation showed, ADVA successfullycombined its vast experience in manufacturingclassic Ethernet-based CPEs with a solid NFV infra-structure based on the compact architecture of theIntel® Xeon® processor D family to construct aunique edge-based NFV solution with value-addedhardware features.

VNF Lifecycle Management

ADVA successfully demonstrated extensive lifecyclemanagement with Brocade vRouter VNF includingOn-Boarding, VNF requirements allocation,Network service creation and Instantiation by usinga standard OpenStack tool.

We started the test session by on-boarding aBrocade vRouter image by using both thecommand-line tool glance as well as via HorizonGUI and verified its presence in the image list, asshown in figure 1. ADVA used a Heat template tosimplify the network service creation which covered

Test Highlights Support for OVS and SR-IOV oper-

ation, SR-IOV halving latency in comparison

Full line rate performance (1Gbit/s) in SR-IOV mode with IPv4 traffic for all packet sizes

Full line rate performance (1Gbit/s) in OVS mode with IPv4/IPv6 traffic mix for packet sizes above 128 bytes

VNF lifecycle management with OpenStack Juno Heat templates

Automated connectivity manage-ment with Modular Layer 2 (ML2) plugin for OpenStack's Neutron

Hardware-based performance assurance functions for network connectivity and VNF hosting do not consume compute resources

Hardware-based precision time synchronization with slave clock independent of the traffic load

EANTC Test Report: ADVA FSP 150 ProVMe Page 1 of 8

all steps needed for the procedure. ADVA claimedthat cloud-init and config drive are supported forVNF configuration. By using cloud-init, all configu-rations could be loaded during the VNF on-boarding process. In the config drive method, theconfiguration would be saved on an additional diskand loaded after the on-boarding process. Limita-tions in configuring the Brocade vRouter VNF withcloud-init could be solved by directly accessing theVM`s console.

Service chaining requires connectivity to be estab-lished among VMs and from VMs to virtual andphysical network interfaces. Connectivity ismanaged by the OpenStack's Neutron. As the FSP150 ProVMe combines virtual switching capabilitywith physical switches, ADVA provides a mecha-nism driver interfacing with the ML2 plugin intoNeutron for configuring end-to-end connectivitythrough virtual and physical switches in an auto-mated way. Our tests revealed some differences inVLAN handling between SR-IOV drivers and theVM, which were solved by a script provided byADVA.

We validated that the newly deployed BrocadevRouter functioned correctly by sending test trafficthrough the instance.Through these proceduresADVA demonstrated the simplicity of Lifecyclemanagement by using standard OpenStack tools.

Forwarding Performance

The forwarding performance can be verified for partic-ular interfaces or for a whole device by measuring the

throughput and delay performance metric when thedata traffic is transmitted into the interface or device athigh load. The goal of our test was to find the perfor-mance of customer premises equipment (FSP 150ProVMe) virtual switch coupled with a Network Func-tion Virtualization Interface Platform based on IntelXeon processor D family.

The test was performed with a series of standardpacket sizes ranging from 64 to 1518 octets based onRFC 2544 section 26.1 and 26.2, as well as a mix ofpacket sizes realistically resembling typical Internettraffic (IMIX). The IMIX used in the test is defined as aproportional mix of several frame sizes as specified inTable 1 below.

Table 1: IMIX Composition

Figure 1: OpenStack VNF Image List

Frame Size [bytes] Weight PercentageIMIX for IPv4

64 3 4.8%100 26 41.3%256 6 9.5%570 5 7.9%

1300 6 9.5%1518 17 27.0%

IMIX for IPv6128 29 46.0%256 6 9.5%570 5 8.0%

1300 6 9.5%1518 17 27.0%


In this series of tests we measured the throughputand latency metric when the data traffic is trans-mitted into the FSP 150 ProVMe at high load. Wewanted to show what base line performance can beexpected from OVS and SR-IOV in NFV setup. Forthe performance verification we tested with a seriesof simple services, with one and two VNFs in achain as described in figure 2. As a result, the totalforwarding performance of the platform dependson how many services had to be operated in orderto create the desired service chain. As in figure 2the test traffic was transmitted through a single VNFas well as a service chain with two VNFs. For thispurpose ADVA used Brocade vRouter as a VNF tocreate a service chain and to forward the IP trafficbetween two interfaces. In this test case we did notanalyze the performance of Brocade vRouter.

ADVA’s FSP 150 ProVMe platform offers two optionsto connect the VMs, namely Open vSwitch (OVS) andSR-IOV (Single-Root I/O Virtualization). OVS providesa fully functional switch implementation in softwareand is capable of stripping the VLAN tag, thereforethe VNF did not require specific VLAN configuration.

On the contrary, SR-IOV bypasses the OVS andconnects the VNFs directly to the physical ports andenables multiple VNFs to share PCI hardwareresources. SR-IOV is designed to have the perfor-mance similar to PCI pass-through, but allows multipleVMs to be connected via same network interface,separated by VLAN. In the SR-IOV setup, the trafficarrives with VLAN tag at the VM. Our SR-IOV testingfocused on IPv4 traffic while ADVA worked onresolving the problem with VLAN tagged IPv6 trafficon the Brocade vRouter. However, we performed the

OVS test with IPv4 and IPv6 traffic. The test scenariosare categorized as follows.

Table 2: Test Scenarios – Forwarding Performance

The test setup consisted of a FSP 150 ProVMe andan Ixia traffic generator. We connected twoEthernet ports to the physical ports and simulatedbidirectional IP test traffic at the target traffic trans-mission rate of 1 Gbit/s between two ports of FSP150 ProVMe using IxNetwork software. By usingRFC2544 test methodology, we found thethroughput value for each of the frame sizes bylocating the maximum traffic rate where no lossoccurs through binary search. At the same time, wemeasured the latency achieved at the maximumbandwidth. The goal of the test was to find theperformance metric of OVS and SR-IOV solutions inADVA’s FSP 150 ProVMe platform.

We achieved full line rate performance (1 Gbit/s)for all scenarios and packet sizes with the soleexception of the 2-VNF setup with OVS and 82-bytepackets. This behavior is within expectations, astraffic consisting of small packets results in higherpacket rate and relative processing overhead.

Parallel to the throughput, we measured the latencyfor each setup. As expected, the latency increaseswith the number of VNFs, as the packets arepassing more segments in the service chain (seefigure 4 and 5). Also as expected, SR-IOV achieves

Figure 2: VNF Chaining – Forwarding Performance

Router

vSwitch vSwitch

Router

Traffic Generator

FSP150 ProVMevSwitchBrocade vRouter VNF

Virtual networksGigabit Ethernet

Router

Physical port Virtual port

Router

Test VNFs vSwitch Traffic1 1x VNF OVS IPv4+IPv62 2x VNF OVS IPv4+IPv63 1x VNF SR-IOV IPv44 2x VNF SR-IOV IPv4


slightly lower latency due to bypassing the OVS.The increased number of interconnections leads toincreased processing time for the switch process.OVS with two VNFs has the highest averagelatency for all the packet sizes; notably it was 224microseconds (μs) at 1024 bytes packet size (seefigure 4). The lowest average latency (46 μs) wasmeasured for SR-IOV with one VNF at 128 bytespacket size.

Figure 5 indicates that OVS with two VNFs has thehighest maximum latency measurement at 1024bytes packet size, which is 868 μs. The lowestmaximum latency is measured for SR-IOV with oneVNF at 82 bytes and 128 bytes packet size, whichis 192 μs.

From figure 4 and 5 we can conclude that OVS hastwice the latency of SR-IOV; with SR-IOV it can beclearly seen that latency scales linearly with thenumber of VNFs.

Hardware-Based Support Functions

In addition to the high performance server, the FSP150 ProVMe provides a wide range of hardware-assisted support functions such as synchronizationand service assurance. Those functions can be acti-vated without requiring compute resources, hence,not creating any negative impact on the perfor-mance of revenue-generating VNFs. This advantagewas shown with a series of tests demonstrating inde-pendence of firewalling with ACLs, port mirroringand synchronization from service performance. Thetest cases looked at the impact of service activationon VNF latency and analyzed any correlationbetween hardware functions and software appli-ances.

Port Mirroring

ADVA’s FSP 150 ProVMe provides traffic mirroringfunctionality, where the entire traffic through aspecific interface can be copied and sent to anotherphysical or virtual interface. For this test, we set upan additional VM running plain Ubuntu and PCIPass-through connection to Ubuntu VM for portmirroring. This test setup consists of one BrocadevRouter, SR-IOV and one Ubuntu VM. Using stan-dard interface statistics and packet capture inUbuntu VM we verified that FSP 150 ProVMemirrors the traffic during the test. At the same timewe compared the test results with standard SR-IOVwith a single Brocade vRouter test setup.

Data transmission was not significantly affected byusing port mirroring at any frame size (see figure6). We confirmed that FSP 150 ProVMe reaches100% throughput while using the port mirroringfeature

Figure 3: Throughput Performance

Figure 4: Average Latency

Figure 5: Maximum Latency


In figure 7, the highest latency for port mirroringwas observed at 1024 byte packet size, which wasincreased by 97 μs and also the value wasincreased by 40 μs at 1280 byte packet size.

Access Control List (ACL)

The Network Interface Device (NID) component ofthe FSP 150 ProVMe platform can apply ACL rulesto the incoming and outgoing traffic. This canprovide a simple but efficient alternative to a full-featured firewall, or help reduce the traffic load byfiltering out undesirable traffic before it reaches theNFVI. The filtering rules are executed in the orderdefined by their priority value and can allow ordeny ranges of MAC addresses, VLANs or IPv4/IPv6 addresses.

In our test, we defined a total of 20 rules, 10 foreach access and network-side interfaces. Theseincluded 3 MAC-based, 2 VLAN-based and 5 IP-based rules. The rules were defined to allow our testtraffic to pass, but not before the entire list was eval-uated. We manually confirmed that the rules werein fact blocking matching addresses and then ran aregular throughput test. This test setup consists of

one Brocade vRouter, SR-IOV and the abovementioned ACL rules. We compared the test resultswith standard SR-IOV with a single Brocade vRoutertest setup.

In figure 8 we can see that the average latency isincreased by nearly 10 μs only at 82 bytes, 1280bytes and 1518 bytes while using NID ACLs and atmost frame sizes data transmission was notaffected.

In figure 9, the maximum latency for NID ACLs wasobserved at 1518 byte packet size, which wasincreased by 100μs and also the value of NID ACLswas increased by 133μs at 1280 byte packet size.We confirmed that FSP 150 ProVMe reaches 100%throughput while using the ACL feature.

Clock Synchronization

FSP 150 ProVMe platform provides synchronizationdelivery and assurance functions using SynchronousEthernet (SyncE) and IEEE1588 Precision TimeProtocol (PTP) to address timing requirements inmobile networks. It can act as a slave, boundaryand master clock. In this test setup we measured the

Figure 6: Port Mirroring - Average Latency

Figure 7: Port Mirroring - Maximum Latency

Figure 8: ACL - Average Latency

Figure 9: ACL - Maximum Latency


accuracy of the slave clock synchronization with anexternal boundary clock and verified that the slaveclock synchronization is not affected by the trafficload. The device provides pulse-per-second (PPS)signal and 10 MHz outputs which we used tomeasure both phase and frequency quality of thePTP synchronization.

Our test bed consisted of two interconnected FSP150 ProVMe devices and a Calnex Paragon-X testequipment. One of the ADVA devices acted as aslave clock and the other as a boundary clock.Paragon-X has a GPS-fed grandmaster clock deviceand a Phase/ Frequency analyzer. All three devicesuse PTP and the grandmaster clock providessynchronization to the boundary clock, which itthen relays to the slave clock. The PTP signalbetween the boundary and the slave clock ran in aseparate VLAN over the same physical link as thetest traffic. The quality of the frequency and phasesynchronization was measured by the Phase/Frequency analyzer. The analyzer received PPS and10-MHz signals from both grandmaster and theslave clock. We used the G.823 E1 SEC andG.8271.1 masks as a requirement for frequencyand phase synchronization and verified the long-term stability of the synchronization by running themeasurement for 6 hours. Figure 10 depicts the testsetup.

We started the test by allowing the boundary clockto acquire a lock via the GPS-fed grandmasterclock. Once it was locked, the slave started to lockto the Boundary Clock. PPS and 10 MHz signals of

the locked slave clock were measured by thePhase/Frequency analyzer with grandmastersignals. We performed the test once with trafficload to verify that PTP synchronization was notaffected by traffic running in parallel. To define thenetwork load we used a network load profile fromITU-T G.8261, test case 13.

The measured average absolute time error of PPSsignal was only -11 ns against the mask G8271accuracy level 4 (±1.5μs) requirements of ±1.1μs.The measured maximum Maximum Time IntervalError (MTIE) value of the PPS signal was 14 ns andwhen traffic was added, it increased only by 44 nsand was still well below the mask.

As well as the measured maximum MTIE value offrequency was nearly 19 ns and during the traffic itincreased only by 43 ns. All the measurementsshow that phase and frequency successfullysynchronized with slave clock and the synchroniza-tion was not affected by the traffic load.

VNF Performance Assurance Functions

The FSP 150 ProVMe provides a set of advancedassurance functions for VNF in-service monitoring.EECPA (Enhanced Ethernet Connection PerformanceAnalysis) provides an integrated traffic generator/analyzer for synthetic performance measurements.EECPA performs measurements between the internalvirtual ports, therefore using different referencepoints than the external Ixia analyzer. Due to ashorter data path which excludes the hardware NID

Figure 10: Test Setup – Clock Synchronization

Traffic Generator

FSP150 ProVMevSwitchBrocade vRouter VNF PTP Daemon

Virtual networksGigabit Ethernet

Router PTP

Physical port Virtual port

Router BoundaryClock

Router SlaveClock

10 MHz signal PPS signal PTP GPS signal

Phase/FrequencyAnalyzer

Paragon X

GrandmasterClock

12:50:00

GPS


component, the measured latency is lower, asexpected.

This case is applicable for one way testing of VM(s)in the embedded or external service while EECPAgenerator and monitor are implemented in theinternal hardware as shown in the figure below.Since EECPA generator and monitor are located inthe same Node test operation and delay measure-ment is tightly coordinated: same timestamp gener-ator is used and test start and test end scheduling issimple.

In this test we used only one stream to generate1Gbit/s IPv4 traffic with 100 bytes frame size. Forthe analysis we used the OVS and SR-IOV setupand measure the latency of EECPA measurement.We confirmed that FSP 150 ProVMe is able togenerate test traffic at 100% throughput andmeasure the performance of the VNFs or ServiceChain it is applied to.

Figures 12 and 13 show examples of the test resultsmeasured by EECPA, showing how FSP 150

ProVMe is able to measure the latency achievedthrough the service chain of VNFs provisioned onthe unit.

About EANTC

EANTC (European AdvancedNetworking Test Center) isinternationally recognized asone of the world's leadingindependent test centers fortelecommunication technolo-gies. Based in Berlin, thecompany offers vendor-

neutral consultancy and realistic, reproducible high-quality testing services since 1991. Customersinclude leading network equipment manufacturers,tier-1 service providers, large enterprises andgovernments worldwide. EANTC's proof ofconcept, acceptance tests and network audits coverestablished and next-generation fixed and mobilenetwork technologies.EANTC AGSalzufer 14, 10587 Berlin, [email protected], http://www.eantc.com/

Figure 11: Test Setup - EECPA

Figure 12: EECPA Latency Test Results

Figure 13: EECPA Test Configuration and Results


Intel and Xeon are trademarks of Intel Corporationor its subsidiaries in the U.S. and/or other coun-tries.


Documents

ADVA FSP 150 ProVMe - eantc.de