June 20–23, 2016 | Berlin, Germany

Achieving Low Latency Network Function with OPNFV

Jiming Sun (Intel)

Tapio Tallgren (Nokia)

Agenda

• Network Latency on NFV

• KVM4NFV and Network Latency

• Low Latency Testing Enhancement on OPNFV

Low network latency is important

• Voice quality for VOIP service - 150ms (tolerable by ears)

• LTE Cloud RAN (Radio Access Network): 1 ms

• - With 5G, the latencies need to be 1/10th of that

• A VNF that receives 10 Gbit/s network traffic of 256 byte packets will receive 3900 packets in 1 ms

• Profit or loss in algorithmic trading requires microsecond latencies

HW

Network Latency and NFV

Open vSwitch VNF1

DPDK Apps

virtio NIC

Linux

HWNIC NIC

vHost

PMD

QEMU

KVMNIC

DPDK Apps

Hypervisor and Network Latency

• Virtualization overhead

• vCPU pre-emption/scheduling

• Virtual switch scheduling/service

• Resource allocation

KVM4NFV Project Introduction

• Join OPNFV on B release

• 3 focus area• Low Latency

• Fast live migration

• Inter-VM communication

• Upstream based effort

Today’s focus

Network Latency with KVM4NFV – B release

• Timer latency (Average: 5 us, Maximum: 15 us)

• Interrupt latency (Maximum: 20 us)

• Maximum guest vCPU pre-emption period (Maximum: 10 us)

• Pre-emption periods in total within 1 ms window (Maximum: 20 us)

Network Latency with KVM4NFV – C release

• Real network application• SR-IOV Packet forwarding

• DPDK Packet forwarding

• Add complexity to the test environment• Stress workload/Noicy neighbor

• Kernel update• From 4.1 LTS to 4.4 LTS

VNF1

DPDK L2 FWD

Linux

HWNIC1 NIC2

TrafficGen

NIC1 NIC2

Packet Forwarding Latency Enhancement

Kernel Average Latency

Maximum Latency

General Kernel

289.6 us 373.9 us

KVM4NFV 19.6 us 101.8 us

• Guest

• NIC devices assigned to guest directly

• DPDK BasicFwd workload

• Host: KVM4NFV

• TrafficGen: 64 byte packet with 10G line rates throughput for 24 hours

VNF1

DPDK Apps

virtio NIC

Hypervisor

Open vSwitch

User Level User Level

virtio NIC

HWNIC NIC

vHost

PMD PMD

vHost

Packet Latency Enhancement (Phy-VM-Phy)

Test Environment:• VM with 4 vCPU• 2 vCPU for data path• PMD and vCPU threads are pinned to isolated CPUs• 2 rx/tx queues per NIC port• 2 rx/tx queues per vHost user port

Testpmd on VNF• 10 minutes testing• 30% line rates and 100% line rates

Packet Latency Enhancement (Phy-VM-Phy)

64 128 256 512 1024 1280 1518

Average (30% ) 14.2 11.2 10.8 10.6 11.4 10 10

Maxium (30%) 31.3 29.9 33.7 29.2 36.1 33 35.8

Average(100%) 25.9 28.8 39.5 46.1 37.7

Maxium(100%) 38.5 103.4 109.5 115 115.2

0

20

40

60

80

100

120

140

Average (30% ) Maxium (30%) Average(100%) Maxium(100%)

Based on 2 hours of tests

HW

CAT and Isolation

Linux KVM

core core

LLCCPU

Memory

VNFStress• LLC (Last Level Cache) is shared

by all Cores on the CPU

• LLC noise impact VNF• Local LLC hit: 40~75 cycles *

• Local DRAM access: 60ns

*http://software.intel.com/sites/products/collateral/hpc/vtune/performance_analysis_guide.pdf

LLC LLC

CAT and Isolation

0

1000000

2000000

3000000

4000000

5000000

6000000

7000000

8000000

0 3 6 9 1215182124273033363942454851545760636669727578818487

Occ

ura

nce

s

Latency

Cyclictest on Idle System

0

200000

400000

600000

800000

1000000

1200000

1400000

1600000

0 3 6 9 1215182124273033363942454851545760636669727578818487

Occ

ura

nce

s

Latency

Cyclictest + stress load with CAT disabled

0

1000000

2000000

3000000

4000000

5000000

6000000

7000000

8000000

0 3 6 9 1215182124273033363942454851545760636669727578818487

Occ

ura

nce

s

Latency

Cyclictest + stress load with CAT

Low Latency Testing on OPNFV

• A automatic latency test system is important• Cover different scenarios

• Catch regression

• But challenges• Complex environment setup

• Openstack or no openstack• Not ready for environment setup

• Analysis not easy with openstack

• Resource requirement• 24 hours testing for each scenario

Target of latency test

• Easy to support different scenarios• Cyclictest->packet forwarding->…..

• Flexible testing scenarios• 24 hours release testing

• Short-term patch testing

• Automatic data publish

Low Latency Test Status

Patch Submission

Artifact Build

Gerrit/Jekins Octopus

Verify or Testing

Job Dispatch

OctopusPharos

Yardstickvsperf

See discussion on https://etherpad.opnfv.org/p/nfv-kvm-test

Data publish

?

Call to Actions

• Participate in KVM4NFV project• Requirements

• Try and provide feedback

• Development

• Collaboration on test workloads• vsperf

• Yardstick