The analysis of Microburs (Burstiness) on Virtual Switch

Copyright 2016 FUJITSU LABORATORIES LIMITED

The Analysis of Microburst (Burstiness) on Virtual Switch

Chunghan Lee Fujitsu Laboratories

09.19.2016

Background

What is Network Function Virtualization (NFV) ?

NFV virtualizes network functions (e.g. Firewall, Load balancer) on IA servers

•The function is called as virtualized network function (VNF)

•Virtual switch is one of major components for virtual network on NFV Infrastructure (NFVI)

•QoS property is also important for NFV

1 Copyright 2014 FUJITSU LABORATORIES LIMITED

An image is from http://www.etsi.org/deliver/etsi_gs/NFV/001_099/002/01.01.01_60/gs_NFV002v010101p.pdf

Virtual network*

*Virtual switch is included in virtual network

Microburst – (1) What is microburst ?

Spikes in shot time period, but causing decreased performance

Impact of microburst on network

Although traffic with QoS property is generated, the spikes can be occurred by massive traffic in very short time period

•This phenomenon cannot catch the ordinary monitoring (SNMP)

•When packet drop is occurred by the spike, it is changed as the microburst


Sudden spikes in throughput

Packet drop !!

SNMP (Coarse-grained)

Packet (Find-grained)

Spike

Microburst – (2)

Impact of microburst on NFVI

Although traffic with QoS property is generated on VNF, the sudden spikes would be occurred due to resource state

Packet loss can be occurred by the spikes (microbursts)


Traffic with QoS

Sudden spikes !!

NFVI (tx) NFVI (rx)

Problem

Microburst on NFVI

Ordinary application measurement cannot catch the microbursts on NFVI

There are queues and buffers between NFVIs

A deep understanding of packet processing issued by virtual switch and Linux kernel is required to clarify the cause of microburst


Traffic with QoS

Sudden spikes !!

NFVI (tx) NFVI (rx)

GOAL

Investigate the occurrence of microburst on NFVI

Find the cause of microbursts on NFVI


Approach

Prepare two types of UDP traffic to observe the microbursts on an OvS bridge

Foreground : Target with QoS property

Background : Occur a lack of CPU resource at kernel

Generate fore/background traffic and capture their packets with tcpdump

Analyze them in packet-level to observe the microburst

Profile kernel functions to clarify the cause of microbursts


Overview of testbed Foreground traffic (UDP)

Sending rate : 2Gbps [Datagram size : 1400 bytes (No fragmentation)]

vport tx queue : default value (0), UDP buffer : default value (200 KBytes)

Background traffic (UDP)

8 vports are used on the same OVS bridge

iperf using UDP mode with 10 parallel flows per vport (iperf option –p)

Sending rate of UDP flow : 1Gbps (configured bandwidth by iperf)


Server and switch spec.

Overview of spec.

All servers have the same spec.

Switch spec.

•Fujitsu SR-X 526 (10G switch)

8

Server type : Fujitsu RX100S7

CPU (4 cores) Intel(R) Xeon(R) CPU E31220 @ 3.10GHz

Memory 16GB (Speed: 1333 MHz)

OS CentOS 7.2

Kernel version 3.18.25

NIC 10G:Intel X710

iperf 2.0.8

Open vSwitch 2.4.0 (release version)

vport queue Default value (txqueuelen : 0)

UDP buffer Default value (200 Kbytes)

Copyright 2016 FUJITSU LABORATORIES LIMITED

Investigation of microburst


Measurement points

6 throughput measurement points

Sender : iperf, Entering OVS, Leaving OVS

Receiver : iperf, Entering OVS, Leaving OVS


iperf User

Kernel

NIC driver

< Sender > < Receiver >

iperf

OVS

(bridge)

NIC driver

OVS

(bridge)

Network

iperf (RX) iperf (TX)

Entering OVS

(tcpdump)

Leaving OVS

(tcpdump)

Throughput measurement points

Entering OVS

(tcpdump)

Leaving OVS

(tcpdump)

Throughput at 6 measurement points Throughput of foreground per second

Leaving OVS (sender), the throughput is decreased

At receiver, the throughput is also decreased at iperf (RX)


< Sender >

< Receiver >

Decreased throughput

Decreased throughput

Throughput of foreground at sender Throughput of foreground per 1 millisecond

Entering OVS, the sending rate is fluctuated due to a lack of CPU

Leaving OVS, sudden spikes in throughput are found

•Throughput with QoS property is changed


< Entering OVS >

< Leaving OVS >

Unit time : 1 millisecond

NIC driver

iperf

OVS (bridge)

< Sender >

Entering

OVS

Leaving

OVS

Network

Packet drop and throughput at receiver A relation between throughput and packet drop

The timing of packet drop is similar to the timing of sudden spikes in throughput at receiver

The overflow of socket buffer is frequently occurred by the sudden spikes (spikes → microbursts)


< Packet drop at socket >

< Leaving OVS >

NIC driver

iperf

OVS (bridge)

< Receiver >

Leaving

OVS

Network

Socket

Packet drop and spacing at receiver A relation between packet drop and packet spacing

The packet spacing with moving average (MA)* is decreased while the number of drop packets on socket buffer is increased


*MA leg : 1000

Packet spacing Burstiness Packet loss

↓ ↑ ↑

↑ ↓ ↓

Microburst

NIC driver

iperf

OVS (bridge)

< Receiver >

Leaving

OVS

Network

Socket

Cause of microburst


Find the cause of microburst

Profiling kernel functions using perf

Profiling rate (sampling rate) is 1 millisecond

Focus on process (iperf with 2Gbps) only for the profiling


Profiling Kernel by perf

Common function call graph

Packet processing at Linux kernel


Net I/F

Enqueue

(qdisc, tx-ring)

TCP/UDP layer (TX)

iperf

OVS (bridge)

IP layer (TX)

Kernel

User

From vport

to OVS

Copy data

From OVS

to qdisc

Measurement points with TCP/IP stack

Leaving OVS, the packet capture is occurred after qdisc


Entering OVS

Leaving OVS Entering OVS

Leaving OVS

Throughput measurement points

Summary Throughput of foreground per second

Leaving OVS (sender), the throughput is decreased

At receiver, the throughput is also decreased at iperf (RX)


< Sender >

< Receiver >

Packet drop at qdisc

Packet drop at socket buffer by microbursts

Conclusion

We investigated the occurrence of microbursts on NFVI A major cause of microburst is packet queuing on qdisc, and

the packet loss at socket buffer on receiver is occurred by the microbursts

At qdisc, the throughput is decreased to 63% and the queue size is not enough to absorb the packets

At socket buffer (rx), the throughput is decreased to 41% and the buffer capacity is also not enough

We found the cause of microbursts using kernel profiling Although the total sending rate is 10 Gbps, qdisc at the sender

is frequently full


Future work

Clarify the cause why qdisc is frequently full although the total sending rate is 10 Gbps

Analyze the profiling results with kernel trace

Modify Linux kernel to change the packet capture point

Extend the experiments

Without OVS (Clarify the overhead of OVS)

With TCP, DPDK OVS, VMs based on vhost


Copyright 2015 FUJITSU LABORATORIES LIMITED 22

Data & Analytics

The analysis of Microburs (Burstiness) on Virtual Switch