Trex Realistic Traffic Generator - download.safeplus.pl Live Berlin 2017/DEVNET... · • Supports 1/10/40 Gb Intel NICs ... • With other NICs (Intel I350, 82599), implemented in

TrexRealistic Traffic Generator

Hanoch Haim – Principal Engineer

DEVNET-1120

• Introduction to Stateful /Stateless

• Current situation and problem

• Model

• High level features

• Examples

• Demo

Agenda

Stateful

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

• Networks include complex L4-7 features, such as• Load Balancer, DPI/AVC, Firewall, NAT

• Requires testing with stateful and realistic traffic mix

Firewall, NATDPI/AVCLB

DEVNET-1120 5

Current Situation


• Traffic generators for realistic traffic are

• Expensive ~$100-500K

• Not scalable for high rates

• Not flexible

• Implication • Limited and late testing

• Different benchmarks and test methodologies

• Real life bottlenecks and design issues

DEVNET-1120 6

What Problem is Being Solved?


• Stateful traffic generator – smart repaly

• Generates, manipulates and amplifies based on templates of real, captured flows – no TCP/IP stack

• High performance: up to 200 Gb/sec

• Low cost: C220M UCS-1RU, Cisco internal • Standard server hardware

• Flexible and Open Software - DPDK, ZMQ, Python libs

• Virtualization

• Easy installation and deployment

DEVNET-1120 7

What is TRex?


Stateful Traffic Generation Model

DEVNET-1120 8


High level software architecture

DEVNET-1120 9

CP

U/S

ha

re m

em

ory

Sh

are

m

em

ory

Inte

l N

IC

DP -thread 1

CP – thread 0

DP -thread 2

Rx – thread 5

IF0Client

IF1Server

IF2Client

IF3Server

DP -thread 3

DP -thread 4

• DPDK/Multi-Threaded• Scales linearly

• ~20 Gb/sec per core

• Supports 1/10/40 Gb Intel NICs

• Flow-based• Fast event scheduler

• Generates flow templates

• Can support 1K templates

• Scales up to 100K clients,1M servers

• Flexible • Client/server generation models• Measures jitter/latency/flow order

• NAT translation/IPv6/Tunnels

• User Interface• Python API

• Benchmark automation

• GUI


Flow Generation

DEVNET-1120 10

Total PPS =

𝑘=0

𝑛

𝐶𝑃𝑆𝑘 × 𝑓𝑙𝑜𝑤_𝑝𝑘𝑡𝑠𝑘

Total CPS =

𝑘=0

𝑛

𝐶𝑃𝑆𝑘

Concurrent flows

=

𝑘=0

𝑛

𝐶𝑃𝑆𝑘 × 𝑓𝑙𝑜𝑤_𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛𝑘

• Example of one flow with four

packets

FIF

FIF

FIF


DNS simple profile example

DEVNET-1120 11


DNS output

DEVNET-1120 12


HTTP & DNS

DEVNET-1120 13


HTTP & DNS

DEVNET-1120 14


Enterprise traffic profile

DEVNET-1120 15

• Includes protocols with

Control/Data dependency • SIP

• RTSP


EMIX - YAML

DEVNET-1120 16


EMIX

DEVNET-1120 17


NAT/FW learning translation/randomization

DEVNET-1120 18

1-Tx-SYN

1-Rx-SYN (learn translation per flow)

2-Tx-SYN-ACK

2-Rx-SYN-ACK (learn)

3-Tx-ACK

3-Rx-ACK (learn seq number randomization)


Client Clustering

DEVNET-1120 19

• To simulate a big network

• Scale of the number of

clients

• For Controller testing


Client Clustering #2

DEVNET-1120 20


• Sampling

• Meta data is added to the flow for order check

DEVNET-1120 21

Flow Order Test


Performance

DEVNET-1120 22


Performance average packet size 600B

DEVNET-1120 23

Stateless


Stateless High level functionality

• High scale –~10M-22MPPS/core

• Support 1/10/25/40/100 Gb/sec interfaces

• Support for multiple traffic profiles per interface

• Profile can support multiple streams, scalable to 10K parallel streams

• Supported for each stream

• Packet template

• Field engine program (src_ip = 10.0.0.1-10.0.0.255)

• Send Mode : Continues/Burst/Multi burst support

DEVNET-1120 25


Stateless High level functionality #2

• Interactive support – GUI/TUI

• Statistic per port

• Statistic per stream (by Hardware)

• Latency Jitter per stream

• Fast Python automation support

• Python 2.7/3.0 Client API

• Python HLTAPI Client API

• Multi-user support

DEVNET-1120 26


Traffic Profile Example

DEVNET-1120 27


Control plane High level

DEVNET-1120 28


Multi User

DEVNET-1120 29


TRex Objects relations

DEVNET-1120 30


One stream with two directions

DEVNET-1120 31

Stateless – traffic profile


Simple Interleaving streams

DEVNET-1120 33


Simple Interleaving streams -profile

DEVNET-1120


Multi burst

DEVNET-1120 35


Multi burst profile

DEVNET-1120 36


Field Engine

DEVNET-1120 37

• Flexible engine to change any field inside the packet

• Examples

– Change TOS 1-20

– Range of client IP 10.0.0.1-10.0.0.254

– Random packet size 64-9k

– Random dest_ip range

– Support any tunnel even not valid packet like

QinQ/GRE/MPLS/Ipv6/UDP/Ipv4/HTTP

• Plan to add even more flexible engine - JITLUA


Instructions

DEVNET-1120 38

https://trex-tgn.cisco.com/trex/doc/cp_stl_docs/api/field_engine.html

Instruction type Description API

STLVmFlowVar Define a variableSTLVmFlowVar(name, init_value=None, min_value=0

, max_value=255, size=4, step=1, op='inc')

STLVmFlowVarRepetableRandom Repeatable random varSTLVmFlowVarRepetableRandom(name, size=4, limit

=100, seed=None, min_value=0, max_value=None)

STLVmTupleGen Tuple generator structSTLVmTupleGen(name, ip_min='0.0.0.1', ip_max='0

.0.0.10', port_min=1025, port_max=65535, limit_

flows=100000, flags=0)

STLVmTrimPktSize Trim the packet size by var name STLVmTrimPktSize(fv_name)

STLVmFixIpv4 Fix IPv4 header checkssum STLVmFixIpv4(offset)

STLVmFixChecksumHw Fix TCP/UDP checksum STLVmFixChecksumHw(l3_offset, l4_offset, l4_typ

e)

STLVmWrMaskFlowVar Write a variable to bitsSTLVmWrMaskFlowVar(fv_name, pkt_offset, pkt_

cast_size=1, mask=255, shift=0, add_value=

0, offset_fixup=0, is_big=True)

STLVmWrFlowVar Write a variableSTLVmWrFlowVar(fv_name, pkt_offset, offset_f

ixup=0, add_val=0, is_big=True)

https://trex-tgn.cisco.com/trex/doc/cp_stl_docs/api/field_engine.html


Field Engine, Syn attack

DEVNET-1120 39


Convert pcap Packet File to One Stream

DEVNET-1120 40

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco PublicDEVNET-1120 41

Pcap File Conversion to Streams


• In this mode pcap in converted to streams and push to TRex server

• It won’t work on a big pcap file

• There is an API version that push server side pcap file

• This version is limited only by server disk size. 1TB pcap file is something that we are using

DEVNET-1120 42

Pcap File Conversion to Profile of Streams #2


• Implemented using hardware assist with Intel

X710/XL710 NIC flow director rules

• With other NICs (Intel I350, 82599), implemented in

software.

DEVNET-1120 43

Per Stream Statistics


Per Stream Statistics - TUI


Per Stream Statistics – Python API


• Base on per stream stats hardware assist

• Forward specific type of packets

• Filter is based on IPV4.ID and IPv6.flow_id

• Software measures latency and jitter resolution is

~usec (not nsec)

DEVNET-1120 46

Per Stream Latency/Jitter


Per Stream Statistics - TUI


Service Mode


• Ping / ARP

• IPv6 ND /Multicast setup

• Scan6 support

• Traffic Capturing

• Functional test

• New protocols support for setup

DEVNET-1120 49

Service Mode


Capture Monitoring – Wireshark Pipe

DEVNET-1120 50

Stateless – Performance


link

Performance Setup

DEVNET-1120 52

trex-tgn.cisco.com/trex/doc/trex_stateless_bench.html


link

Performance XL710 MPPS/Core

DEVNET-1120 53


Stateless – Automation


Simple example

DEVNET-1120 55


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

DEVNET-1120 56

CiscoLive.com/Online


Continue your Education

DEVNET-1120 57

• Stateless manual

• TRex documents Index

• GitHub

• DevNet zone

http://trex-tgn.cisco.com/trex/doc/trex_stateless.html

http://trex-tgn.cisco.com/trex/doc/

https://github.com/cisco-system-traffic-generator/trex-core

https://developer.cisco.com/site/trex/

Q & A

Thank You

Backup

SR-IOV support


SR-IOV vs vSwitch

63DEVNET-1120

VMM

vNIC vNIC

vSwitch

NIC PF

VMM

vNIC vNIC

HW SwitchNIC PF

VFVF

SR-IOVvSwitch


SR-IOV XL710/X710

64DEVNET-1120

VMM

vNIC vNIC

HW SwitchNIC PF

VFVF

Pool of TRex

TRex TRex

vNIC vNIC

HW Switch

NIC PF

VFVF

Pool of CSR

CSR CSR

Catalist 36xx

25/50/100GbE support


• 100/50/25/40/1 GbE speeds

• PCIe16 v3

• Small foot print

• Low cost - 800$

• SR-IOV support

• 10-60K TCAM rules – not perfect rule for counters

DEVNET-1120 66


ConnectX-4 DPDK performance

DEVNET-1120 67

• UDP 64B

• TX ~95 MPPS

• Rx 50 MPPS

• IMIX

• 90% line-rate

• Cycles/Pkt high +50% relative to Intel


FPGA Base – 400Gbe

68DEVNET-1120

• FPGA base

• Accurate

IPG/latency

• Scale to 400GbE

without DP SW

• ~30K$ –

4x100GbE

• Better Statistic in

Rx side

• Less CPU

resource

• Manufactured by

Cisco VIC support


Cisco VIC 13xx support

70DEVNET-1120

• Contribution from DPDK VIC team

• Supported on both UCS C-series and B-series (blade

server)

• PCIe16 - 2x-40GbE ports

• Can be shared by VM using Cisco vNIC (using CIMC) - it

is not SR-IOV

• Only the 13xx series, Cisco adapter is supported

Demo Statful


Demo routing

DEVNET-1120 72

Cisco ASR 1013 ESP100

100Gb/sec 13RU - 4KW

UCS-220M2 32GB 2x8 cores 2Ghz

2x82559 NIC (4x10Gb/sec)

0.4KW 1RU , 2K$


-Per port stats table

ports | 0 | 1 | 2 | 3

-----------------------------------------------------------------------------------------

opackets | 71602327 | 82914082 | 71601627 | 82913259

obytes | 23990692045 | 67783397687 | 23990683213 | 67783032819

ipackets | 82914006 | 71602322 | 82913203 | 71601615

ibytes | 67783068258 | 23986183633 | 67782693539 | 23986134949

Tx Bw | 3.68 Gbps | 8.84 Gbps | 3.68 Gbps | 8.84 Gbps

Cpu Utilization : 41.6 % 15.0 Gb/core

Platform_factor : 1.0

Total-Tx : 25.04 Gbps Nat_time_out : 0

Total-Rx : 25.03 Gbps Nat_no_fid : 0

Total-PPS : 5.21 Mpps Total_nat_active: 33

Total-CPS : 103.16 Kcps Total_nat_open : 6697955

Active-flows : 90502 Clients : 248 Socket-util : 0.5795 %

Open-flows : 6713475 Servers : 5368 Socket : 90502 Socket/Clients : 364.9

-Latency stats enabled

Cpu Utilization : 7.2 %

if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window

| , , check, , latency(usec),latency (usec) ,(usec) ,

--------------------------------------------------------------------------------------------------------------

--

0 | 65649, 65648, 647320, 0, 61 , 144, 13 | 111 130 138 107 127 110

1 | 65648, 65649, 552181, 0, 37 , 62, 3 | 42 42 43 39 38 39 38

2 | 65649, 65648, 660627, 0, 57 , 144, 11 | 103 139 130 97 126

3 | 65648, 65649, 562860, 0, 37 , 55, 3 | 39 41 39 39 46 39 39

-Rx Check stats enabled

------------------------------------------------------------------------------------------------------------

rx check: avg/max/jitter latency, 63 , 176, 18 | 130 158 159 138 155 175 136

active flows: 647, fif: 52193, drop: 0, errors: 0

------------------------------------------------------------------------------------------------------------


Configure NAT & Firewall


-Global stats enabled

Cpu Utilization : 34.9 % 15.9 Gb/core

Platform_factor : 1.0

Total-Tx : 22.24 Gbps Nat_time_out : 3910462

Total-Rx : 22.24 Gbps Nat_no_fid : 735

Total-PPS : 4.87 Mpps Total_nat_active: 46

Total-CPS : 102.79 Kcps Total_nat_open : 11287417

Active-flows : 79813 Clients : 248 Socket-util : 0.5110 %

Open-flows : 11313545 Servers : 5368 Socket : 79813 Socket/Clients : 321.8

-Latency stats enabled

Cpu Utilization : 6.7 %

if| tx_ok , rx_ok , rx ,error, average , max , Jitter , max window

| , , check, , latency(usec),latency (usec) ,(usec) ,

--------------------------------------------------------------------------------------------------------------

--

0 | 110281, 110279, 714052, 0, 204 , 14313, 16 | 807 1276 3439 935 850 5175

1 | 110280, 110280, 644514, 0, 91 , 10907, 5 | 802 653 1344 717 701 3436

2 | 110281, 110280, 720630, 0, 201 , 14355, 14 | 808 1194 3357 972 822 5177

3 | 110280, 110281, 645768, 0, 90 , 10491, 3 | 800 642 1349 706 699 3279

-Rx Check stats enabled

------------------------------------------------------------------------------------------------------------

rx check: avg/max/jitter latency, 180 , 14785, 407 | 2321 2546 4229 1627 1912 5275

active flows: 690, fif: 57593, drop: 14179, errors: 14552

------------------------------------------------------------------------------------------------------------

DEVNET-1120 76



TRex on your laptop (virtualbox)

DEVNET-1120 78


TRex DP queues/cores #1 DP

DEVNET-1120 79

Flow

Director


TRex DP queues/cores #2 DP

DEVNET-1120


TRex CP-DP messaging

DEVNET-1120 81

• One to many

• No locks


TRex RX-DP messaging

DEVNET-1120 82

• One to many

• No locks


Stateful vs Stateless

DEVNET-1120 83

Feature Stateless Stateful

Flow base No Yes

NAT No Yes

Tunnel Yes Some are supported

L7 App emulation No Yes

Any type of packet Yes No

Latency/Jitter Per Stream Per port/Per flow sample


Interactive Console

DEVNET-1120 84

#load the trex as a server for interactive mode

$sudo ./t-rex-64 –i

#connect to the server from any server ( Python 2/3.4)

$./trex-console

#start traffic on all port

>start -a -m 1 -f stl/imix_1pkt.py

#pause traffic on all port

>pause -a

#resume traffic on all port

>resume -a

#stop traffic on all port

>stop -a

#show dynamic statistic

>tui

#show port statistic

>stats –p

#clear statistic

>clear

#show stream statistic

>streams

Shell

Console


Interactive TUI

DEVNET-1120 85


Multiple Clients Example

DEVNET-1120 86


Multiple Clients Profile


Teredo Tunnel (Ipv6 Over IPv4

DEVNET-1120 88


link

• XL710 support line rate

from 128B

Performance XL710 Line%

DEVNET-1120 89



link

Performance XL710 gbps/core

DEVNET-1120 90



Demo

DEVNET-1120 92

Cisco ASR 1013 ESP100

100Gb/sec 13RU - 4KW

UCS-220M2 32GB 2x8 cores 2Ghz

2x82559 NIC (4x10Gb/sec)

0.4KW 1RU , 2K$finalized the GUI

Documents

Trex Realistic Traffic Generator - download.safeplus.pl Live Berlin 2017/DEVNET... · • Supports 1/10/40 Gb Intel NICs ... • With other NICs (Intel I350, 82599), implemented in