Scalable high-speed packetcaptureUsing OpenFlow and Intel DPDK

Wouter de Vries

RIPE71 - Bucharest, Romania

Who am I?

Wouter de VriesPh.D. student

Design and Analysis ofCommunication SystemsUniversity of Twente

Scalable high-speed packet capture November 13, 2015 2 / 24

DACSDesign and Analysis of Communication Systems

Introduction

We want to capture large-scale DDoS attacks withoutsignificant packet loss, why?

I Mitigation is hardI In-depth analysis could provide valuable insights

Other uses of high-speed packet capture:I Intrusion detectionI Monitoring (start your own NSA!)

Scalable high-speed packet capture November 13, 2015 3 / 24

The Problem

The total bandwidth of The InternetTM is ever increasing.

Table: Cisco Visual Networking Index 2015

Year 2014 2015 2016 2017 2018 2019PB per Month 59,8 72,4 88,4 109,0 135,5 168,0

Scalable high-speed packet capture November 13, 2015 4 / 24

The Problem

The total bandwidth of The InternetTM is ever increasing.

Table: Cisco Visual Networking Index 2015

Year 2014 2015 2016 2017 2018 2019PB per Month 59,8 72,4 88,4 109,0 135,5 168,0

Scalable high-speed packet capture November 13, 2015 4 / 24

The Problem

In order to analyze real-world traffic, the capture methods needto evolve.At speeds in excess of 10 Gbit/s things start to get difficult:

I ≥ 14.8 million packets per secondI Only a few clockcycles per packetI Storing ≥1.25 Gigabytes per second

Scalable high-speed packet capture November 13, 2015 5 / 24

The Problem

In order to analyze real-world traffic, the capture methods needto evolve.At speeds in excess of 10 Gbit/s things start to get difficult:

I ≥ 14.8 million packets per second

I Only a few clockcycles per packetI Storing ≥1.25 Gigabytes per second

Scalable high-speed packet capture November 13, 2015 5 / 24

The Problem

In order to analyze real-world traffic, the capture methods needto evolve.At speeds in excess of 10 Gbit/s things start to get difficult:

I ≥ 14.8 million packets per secondI Only a few clockcycles per packetI Storing ≥1.25 Gigabytes per second

Scalable high-speed packet capture November 13, 2015 5 / 24

The Problem

In order to analyze real-world traffic, the capture methods needto evolve.At speeds in excess of 10 Gbit/s things start to get difficult:

I ≥ 14.8 million packets per secondI Only a few clockcycles per packetI Storing ≥1.25 Gigabytes per second

Scalable high-speed packet capture November 13, 2015 5 / 24

Goal

A scalable system that is able to capture and generatepackets at high speed (e.g. ≥40 Gbit/s)

Scalable high-speed packet capture November 13, 2015 6 / 24

Proposal

I Use DPDK (Data Plane Development Kit) to maximizesingle machine performance.

I Use OpenFlow-switches to distribute traffic over multiplemachines

Scalable high-speed packet capture November 13, 2015 7 / 24

Proposal

I Use DPDK (Data Plane Development Kit) to maximizesingle machine performance.

I Use OpenFlow-switches to distribute traffic over multiplemachines

Scalable high-speed packet capture November 13, 2015 7 / 24

Proposal - Overview

Openflow Switch

Capture machine

Capturemachine

Capturemachine

Capturemachine

40 Gb/s 10 Gb/s

10 Gb/s

10 G

b/s

Recombine (offline)

pcap-file

splits traffic

Scalable high-speed packet capture November 13, 2015 8 / 24

Implementation - What is DPDK?

The Data Plane Development Kit is a library for fast packetprocessing

Main features:I Zero-CopyI Fast buffersI Designed for

multicore

Zero-copy allows the networkhardware to directly copy data tomemory buffers using DMA

Scalable high-speed packet capture November 13, 2015 9 / 24

Implementation - What is DPDK?

The Data Plane Development Kit is a library for fast packetprocessing

Main features:I Zero-CopyI Fast buffersI Designed for

multicore

Fast and thread-safe implementationsof (ring) buffers making developmentof multithreaded applications mucheasier

Scalable high-speed packet capture November 13, 2015 9 / 24

Implementation - What is DPDK?

The Data Plane Development Kit is a library for fast packetprocessing

Main features:I Zero-CopyI Fast buffersI Designed for

multicore

Has been designed from the groundup to support multiple cores, eachthread runs on its own core

Scalable high-speed packet capture November 13, 2015 9 / 24

Implementation - Using DPDK

Capture 64-byte packets at 10 Gbit/s (1.25 GB/s or 1 DVDevery 4 seconds) in PCAP-format on commodity hardware.What to do with all this data?

Scalable high-speed packet capture November 13, 2015 10 / 24

Implementation - Adding compression

gzip bzip2 lzma xz lzo

Compression time

Method

Tim

e (s

)

010

2030

4050

Compressing the linux kernel to a ram disk.Source: http://catchchallenger.first-world.info

Scalable high-speed packet capture November 13, 2015 11 / 24

Intermediate results

I Using compression specially crafted 64-byte packets canbe captured at line-rate on a single conventional HDDusing 3 cores

I Generating packets at line-rate (10 Gbit/s) is possibleusing a single core

Scalable high-speed packet capture November 13, 2015 12 / 24

Implementation - What is OpenFlow?

”OpenFlow allows direct access to and manipulationof the forwarding plane of network devices such asswitches and routers”

— Open Networking Foundation

Scalable high-speed packet capture November 13, 2015 13 / 24

Implementation - What is OpenFlow?

Controller

Openflowswitch

SW

HW

securechannel

Flow table

Scalable high-speed packet capture November 13, 2015 14 / 24

Implementation - OpenFlow

We need to define something that we split the traffic on.Possible candidates:

I Source port for TCP/UDP (allows mask on Open vSwitch)I IP-address (allows mask)I Equal-Cost Multi-Path (ECMP) routing algorithms

Scalable high-speed packet capture November 13, 2015 15 / 24

Implementation - UDP

0

1000

2000

3000

4000

5000

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Stan

dard

Dev

iatio

n

Rela

tive

Stan

dard

Dev

iatio

n

Modulo

Splitting traffic by UDP source port (2167847 packets)

SD RSD

Data: Random DRDoS attack PCAP from simpleweb.org

Scalable high-speed packet capture November 13, 2015 16 / 24

Implementation - Example flow table

Bitmask on last two bits of UDP source port

Scalable high-speed packet capture November 13, 2015 17 / 24

Implementation - IP address

4096

8192

16384

32768

65536

131072

262144

524288

1.04858x106

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Stan

dard

Dev

iatio

n

Rela

tive

Stan

dard

Dev

iatio

n

Modulo

Splitting traffic by last octet of IP address (2167847 packets)

SD RSD

Data: Random DRDoS attack PCAP from simpleweb.org

Scalable high-speed packet capture November 13, 2015 18 / 24

Implementation - IP address

4096

8192

16384

32768

65536

131072

262144

524288

1.04858x106

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0

0.01

0.02

0.03

0.04

0.05

Stan

dard

Dev

iatio

n

Rela

tive

Stan

dard

Dev

iatio

n

Modulo

Splitting traffic by MD5 hash of IP address (2167847 packets)

SD RSD

Data: Random DRDoS attack PCAP from simpleweb.org

Scalable high-speed packet capture November 13, 2015 19 / 24

Implementation - ECMP

Equal-Cost Multi-Path routing is used to balance traffic overmultiple links that have the same cost.

I ECMP Algorithm is not defined by OpenFlowI Result: ECMP implementation varies by vendor

The definition of ECMP is a great match to our problem

Scalable high-speed packet capture November 13, 2015 20 / 24

Current state

I For some types of traffic splitting is easier than othersI On-going work to find a generic way to balance flowsI ECMP is promising, depending on the implementation by

the vendor

Scalable high-speed packet capture November 13, 2015 21 / 24

Conclusion

I Using DPDK allows line-rate packet capture on 10 Gbit/sI Using OpenFlow-compatible switches has the potential to

scale the capture speed horizontallyI Combined, these two technologies allow us to capture

≥40 Gbit/s

Scalable high-speed packet capture November 13, 2015 22 / 24

Open-source

I DPDK-based packet capture tool (DPDKcap):https://github.com/woutifier/dpdkcap

TRY THIS AT HOME

Scalable high-speed packet capture November 13, 2015 23 / 24

Questions

Thank you for your attention!Questions and/or comments are welcome!

Scalable high-speed packet capture November 13, 2015 24 / 24