PLNOG 13: Piotr Głaska: Quality of service monitoring in IP networks

Monitorowanie jakości usług w sieciach IP

Piotr Głaska

Kraków, 29.09.2014

1

Agenda

NQA – Network Quality Analysis

iPCA – Packet Conservation Algorithm for Internet

AtomEngine

2

NQA Working Principle

Client: initiates test, gathers statistics

Server: responds to the test initiated by client

Test results can be viewed through command line, SNMP, can be

uploaded by FTP, can generate logs, alarms and actions

3

Types of NQA tests

Link Layer

IP

ICMP ping, traceroute, jitter

ARP Ping, MAC ping

MPLS

LSP ping, traceroute, jitter

HTTP DNS lookup

FTP Link, download time

DNS

VOIP delay & jitter

Video delay & jitter

DHCP TCP

Three-way handshake

Multicast

Multicast traceroute

Jitter, Echo

UDP

4

Configuring ICMP Test

HuaweiA> system-view

[HuaweiA] nqa test-instance admin icmp

[HuaweiA-nqa-admin-icmp] test-type icmp

[HuaweiA-nqa-admin-icmp] destination-address ipv4 10.1.1.2

[HuaweiA-nqa-admin-icmp] start now

5

Configuring Jitter Test

<HuaweiB> system-view

[HuaweiB] nqa-server udpecho 10.1.1.2 9000

<HuaweiA> system-view

[HuaweiA] nqa test-instance admin jitter

[HuaweiA-nqa-admin-jitter] test-type jitter

[HuaweiA-nqa-admin-jitter] destination-address ipv4 10.1.1.2

[HuaweiA-nqa-admin-jitter] destination-port 9000

[HuaweiA-nqa-admin-jitter] start now

6

IP SLA and NQA interoperability

NQA can work with IP SLA as a responder for UDP Echo and UDP Jitter tests

[Huawei] ip nqa-compatible responder [vpn-instance vpn-instance-name] enable

[Huawei] ip nqa-compatible auto

7

Router A Router B

Router C Router D

GE1/0/0

GE1/0/0

GE1/0/0

GE1/0/0

GE2/0/0 GE2/0/0

GE2/0/0

GE2/0/0

<RouterA> system-view

[RouterA] nqa test-instance user test

[RouterA-nqa-user-test] test-type icmp

[RouterA-nqa-user-test] destination-address ipv4 4.1.1.2

[RouterA-nqa-user-test] start now

[RouterA] interface gigabitethernet2/0/0

[RouterA-GigabitEthernet2/0/0] standby track nqa user test

Run the display nqa results test-instance user test command.

Interface Backup and NQA

8

Switch

Router A Router C

Router B Router D

GE1/0/0 10.1.1.1/24

NQA agent

GE2/0/0 192.168.1.1/24

GE1/0/0 192.168.1.2/24

GE2/0/0 20.1.1.1/24

GE1/0/0 10.1.1.2/24

GE2/0/0 192.168.2.1/24

GE1/0/0 192.168.2.2/24

GE2/0/0 30.1.1.1/24

Master

Backup

VRRP Backup Group Virtual IP Address: 10.1.1.10

Host A

VRRP Backup Group with NQA

This mechanism enables the VRRP backup group to monitor the link connecting the master to the

external network. If the link fails, hosts on a LAN cannot access an external network through the

master router. NQA detects this fault and notifies VRRP. The VRRP backup group lowers the master

router's priority by a configured value. The backup router with the highest priority will become the

new master router and take over traffic.

9

DHCP Pool with NQA

[Huawei] ip pool p1

[Huawei-ip-pool-p1] excluded-ip-address 10.1.1.1 10.1.1.100

[Huawei-ip-pool-p1] lock track nqa admin dhcptest

10

DNS Proxy with NQA

[Huawei] dns resolve

[Huawei] dns server 10.1.1.2 track nqa admin localdns

[Huawei] dns server 20.1.1.2 track nqa admin remotedns

[Huawei] dns proxy enable

11

3G/LTE Modem recovery with NQA

[Huawei] interface cellular 0/0/0

[Huawei-Cellular0/0/0] modem auto-recovery track nqa user test

[Huawei-Cellular0/0/0] modem auto-recovery track action { plmn-search |

modem-reboot } fail-times times

12

Adaptive Traffic Shaping with NQA

[Huawei] qos adaptation-profile gts1

[Huawei-qos-adaptation-profile-gts1] rate-range low-threshold 128 high-threshold 512

[Huawei-qos-adaptation-profile-gts1] rate-adjust step 32

[Huawei-qos-adaptation-profile-gts1] rate-adjust loss low-threshold 20 high-threshold 30

[Huawei-qos-adaptation-profile-gts1] track nqa admin jitter1

[Huawei] interface gigabitethernet 1/0/0

[Huawei-GigabitEthernet1/0/0] ip address 192.168.1.2 255.255.255.0

[Huawei-GigabitEthernet1/0/0] qos gts adaptation-profile gts1

[Huawei-GigabitEthernet1/0/0] traffic-policy p1 outbound

When configuring an NQA test instance, ensure that NQA packets enter high-priority queues so that they are treated preferentially when

the link is congested.

13

NQA for Static Routes

ip route-static 172.16.7.0 255.255.255.0 172.16.3.2 track nqa user test

ip route-static 172.16.7.0 255.255.255.0 172.16.4.2 preference 100

nqa test-instance aa bb

test-type icmp

destination-address ipv4 172.16.1.2

frequency 3

probe-count 1

start now

14

Policy Based Routing with NQA

acl number 2000 rule 10 permit source 192.168.1.0 0.0.0.255

traffic classifier vlan10

if-match acl 2000

traffic behavior vlan10

redirect ip-nexthop 192.168.3.2 track nqa admin vlan10

traffic policy vlan10

classifier vlan10

behavior vlan10

interface GigabitEthernet1/0/0

ip address 192.168.1.1 255.255.255.0

traffic-policy vlan10 inbound

15

Smart Policy Routing with NQA

[Huawei] smart-policy-route

[Huawei-smart-policy-route] prober ethernet 1/0/0 nqa admin nqa1

[Huawei-smart-policy-route] prober ethernet 2/0/0 nqa admin nqa2

[Huawei-smart-policy-route] link-group group1

[Huawei-smart-policy-route-link-group group1] link-member ethernet 1/0/0

[Huawei-smart-policy-route] link-group group2

[Huawei-smart-policy-route-link-group group2] link-member ethernet 2/0/0

[Huawei-smart-policy-route] service-map map1

[Huawei-smart-policy-route-service-map-map1] match acl 3000

[Huawei-smart-policy-route-service-map-map1] set link-group group1

[Huawei-smart-policy-route-service-map-map1] set link-group group2 backup

16

LTE APN Tracking with NQA

Example: DSVPN based on 3G/LTE dialup status

DSVPN – Dynamic Smart VPN, dynamic VPN based on NHRP and mGRE

17

3G/LTE APN Tracking with NQA

interface Cellular0/0/0

apn-profile orange priority 200 track nqa admin tunnel0/0/1 admin tunnel0/0/2

apn-profile tmo priority 150 track nqa admin tunnel0/0/3 admin tunnel0/0/4

apn profile orange

apn internet

sim-id 1

apn profile tmo

apn internet

sim-id 2

18

interface Tunnel0/0/1

ip address 172.10.1.2 255.255.255.0

rip metricin 1

tunnel-protocol gre p2mp

source Cellular0/0/0

gre key cipher @%@%.'YF3l/T'GtCF,$NT-<$~5U]@%@%

nhrp authentication cipher %@%@Z1jU$i^[f:xiYUF|Dhj%

nhrp registration interval 20

nhrp entry 172.10.1.1 202.10.1.2 register track apn orange


ip address 172.10.2.2 255.255.255.0

rip metricin 7

rip metricout 7



gre key cipher @%@%f94gE3y!0=%Ba0Y-cSR3~6&<@%@%

nhrp authentication cipher %@%@HP>P#8z<G#*9<7A70!YUG~


nhrp entry 172.10.2.1 202.10.1.10 register track apn orange


ip address 172.10.3.2 255.255.255.0

rip metricin 4

rip metricout 4



gre key cipher @%@%r*crMiQ/b!gLFF~sj}qO~5@f@%@%

nhrp authentication cipher %@%@Q2atQl+%C51rQRSVB


nhrp entry 172.10.3.1 202.10.1.6 register track apn tmo


ip address 172.10.4.2 255.255.255.0

rip metricin 10

rip metricout 10



gre key cipher @%@%<&-+=09yzL]g'*;V)E|~~7"a@%@%

nhrp authentication cipher %@%@oB|n3,7,eP]jh)/KzuN~QOa


nhrp entry 172.10.4.1 202.10.1.14 register track apn tmo

Spoke router tunnels configuration

19

IVPN – Intelligent VPN

20

IVPN – Intelligent VPN ivpn-proposal p1

encapsulation gre

source Dialer1 destination 202.1.1.2 bandwidth up 1024 down 8192

track nqa admin dsl

source Cellular0/0/0 destination 202.1.1.2 bandwidth up 15000 down

30000 track nqa admin lte

service youtube id 1

schedule-type priority

match app-protocol youtube

source Dialer1


cmi-method D/2+ J x 2 + L

cmi-threshold cmi 8500 delay 1000 jitter 500 loss 20

service exchange id 2

schedule-type overload

match app-protocol ms_exchange


source Dialer1


ip address 172.10.1.1 255.255.255.0

tunnel-protocol ivpn p2p

ivpn-zone 1

ivpn-proposal p1

Hub configuration:

ivpn-proposal p1

encapsulation gre

service s1 id 1

match app-protocol youtube

service exchange id 2

match app-protocol ms_exchange


ip address 172.10.1.2 255.255.255.0

tunnel-protocol ivpn p2mp

ivpn-zone 1

ivpn-proposal p1

Default CMI method: CMI = 9000 - (D + J + L)

Default CMI, delay, jitter and packet loss thresholds

are 0, 5000 ms, 3000 ms and 1000‰

21

Scalability

Most of NQA tests are processed by the main core on AR G3 routers. They

run with low priority, so there is little impact on CPU

UDP jitter detection is of milliseconds level and there is larger impact on

CPU. Forwarding cores can support this test and enhance performance of

sending and receiving packets, reducing impact on the main core

UDP Jitter tests can be hardware-based and processed by line cards (LPU)

22

Hardware-based NQA on AR G3 routers

Reduces the interval for sending packets.

The minimum interval for sending packets can be 10 ms.

Increases the number of concurrent test instances (up to 6000) and test

packets per second (up to 2000)

Improves the accuracy of delay and jitter calculation

From miliseconds to microseconds level

[Huawei] nqa test-instance user test

[Huawei-nqa-user-test] test-type jitter

[Huawei-nqa-user-test] hardware-based enable

[Huawei-nqa-user-test] timestamp-unit microsecond

23

Hardware-based IP SLA on ASR1K routers

Use QFP for timestamping

ip sla 1

udp-jitter 192.0.2.134 5000 num-packets 20

request-data-size 160

tos 128

frequency 30

precision microseconds

optimize timestamp

24

iPCA

25

iPCA Concept

Packet Conservation Algorithm for Internet (iPCA) is developed by Huawei. iPCA implements packet loss monitoring and

fault location for connectionless IP networks by coloring real service packets and partitioning a network. It allows a

network to perceive service quality and quickly locate faults. In addition, iPCA breaks the limitation of traditional

measurement technologies.

L2+L3 mixed

network

iPCA

P2P, MP2MP

Performance monitoring based on real

service packets

Question iPCA

Who lost the packets?

Where were the packets lost?

When were the packets lost?

Monitors the service flows with five

specified attributes based on domains or

link segments, and determines the type of

services with packets lost.

Partitions the network into multiple

domains, provides device-level, link-level,

and network-level monitoring, and

automatically locates faults.

Monitors real service flows, and sends

alarms to the administrator immediately

when faults occur.

26

Topology-Centric Configuration and Monitoring

27

Network-Level Measurement View

28

Select building 6 in Warsaw and building 1 in Krakow in the topology view to create a network-level

measurement task. eSight can discover service path, display each agile switch on the path, and show

packet loss on the path.

1 Perform hop-by-hop path measurement in the

network-level view.

2 Display the service forwarding path and

agile switches.

3 Show packet loss statistics on egress node.

Click a device on the path to

show real-time packet loss

statistics on the device

Hop-by-Hop Measurement on Unicast IP Service Path

29

Based on Real Traffic

Traditional quality measurement technologies send

simulated detection packets on the network. The simulated

detection packets are different from real service packets in

sizes and frequencies. Therefore, simulated detection

packets cannot reflect real service quality, and occupy

bandwidth.

Service packets

Simulated detection packets Service packets

iPCA colors real service packets, which do not occupy additional

bandwidth.

30

Packet Loss Measurement Between Two Points

The transmitter colors (sets to 1) and decolors (sets to 0) a characteristics bit in service packets periodically to divide the service flow into

multiple intervals.

The counters and measurement points are configured on the transmitter and receiver, and the number of sent and received packets is counted

based on intervals.

TX[i] and RX[i] packets are sent to the MCP. The MCP identifies the packets, compares the number of sent packets with the number of

received packets in the same interval, and obtains the number of lost packets.

Calculate the measurement result: In the interval i (packet sending interval and receiving interval), the number of lost packets = TX[i] - RX[i].

Interval i Interval i+1

Measurement point in sending direction Measurement point in receiving direction

Service

packet

flow

Transmitter

Interval i

Interval i+1

Receiver

1 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 1 1 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 1

Send: A target flow is divided into

consecutive measurement intervals,

and the number of packets (TX[i])

sent in each interval is counted.

Receive: Identify the measurement intervals and

count the number of packets received in an interval

(RX[i]). Packet unsequencing issue should be

noticed.

31

0 1 2 30 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

Type of ServiceFlags

Options Padding

Version IHL Total LengthIdentification Fragment Offset

Time to Live Protocol Header ChecksumSource Address

Destination Address

Color Bit Selection

The reserved bit in IPv4 packet header can be used as the color bit, for example, bit 0 in the

Flags field or one of bits 3-7 in ToS field.

In iPCA device-level measurement, bit 0 in the Flags field is used as the color bit by default.

In iPCA network-level measurement, bit 6 in the ToS field is used as the color bit by default.

32

iPCA Logical Topology The iPCA system consists of NMS, MCP, DCPs, and TLPs, which have the

following responsibilities:

NMS: provides GUI

Issues commands to configure measurement instances.

Obtains real-time statistics and historical data from MCP, and displays

measurement results.

TLP (Target Logical Port):

Executes iPCA measurement tasks, and corresponds to a logic interface on

network device

Colors and measures target service flows periodically.

Reports statistics in each interval to DCPs.

DCP (Data Collecting Point):

Manages and controls TLPs (configures and issues ACL rules to TLPs).

Collects statistics from TLPs.

Reports statistics to the MCP.

MCP (Measurement Control Point):

Collects statistics from DCPs.

Summarizes statistics and calculate results.

Reports measurement results to the NMS.

DCP

MCP

DCP

TLP

TLP

TLP

TLP

eSight

Management data

Measurement data report

Real service flow

33

Measurement Principle

Measured system (device/link/carrier's

network/service path)

System core

Internally

terminated Internally

generated

iPCA quality measurement mechanism: A measured system is in the normal state if the

following condition is met:

Number of packets arriving at the system + Number of internally generated packets = Number

of packets leaving the system + Number of packets internally terminated by the system

If this condition is not met, some packets have been dropped in the system.

Packets arriving

at the system Packets leaving

the system

34

Device-Level Packet Loss Measurement

Measurement domain: All ENP cards and SFUs form a packet

conservation domain (excluding the CPU and non-ENP cards).

Object: All incoming and outgoing IP unicast flows of the

measurement domain.

Measurement interval: 10 seconds

Alarm: When the packet loss ratios in five consecutive intervals

exceed 5%, the device sends an alarm to the NMS.

When the packet loss ratios in five consecutive intervals fall

below 1%, the device sends a clear alarm to the NMS.

Incoming flows Outgoing flows

CPU Chassis

Non-ENP

SFU 1

Chassis

Ingress TLP Egress TLP

CPU

Non-ENP

card ENP card 1

C1-1

C1-2 C1-3 C1-4 C1-5 C1-6 C1-7

C1-8 C1-9 C1-10

ENP card 2

C2-1

C2-1 C2-3 C2-4 C2-5 C2-6 C2-7

C2-8 C2-9 C2-10

SFU 2

Number of packets from other devices to ENP cards: C1-1 and C2-1

Number of packets from CPU to ENP cards: C1-5 and C2-5

Number of packets from non-ENP cards to ENP cards: C1-7 and C2-7

The number of packets entering the measurement domain Cin = C1-1 + C2-1 + C1-5 + C2-5 + C1-7 +

C2-7

Number of packets from ENP cards to CPU: C1-2 and C2-2

Number of packets from ENP cards to non-ENP cards: C1-4 and C2-4

Number of packets from CPU to ENP cards and then other devices: C1-8 and C2-8

Number of packets from ENP card to ENP card and then other devices: C1-9 and C2-9

Number of packets from non-ENP card to ENP card and then other devices: C1-10 and C2-10

Number of packets leaving the measurement domain Cout = C1-2 + C2-2 + C1-4 + C2-4 + C1-8 + C2-

8 + C1-9 + C2-9 + C1-10 + C2-10

Number of lost packets = Cin - Cout

Measurement

domain

35

Link-Level Packet Loss Measurement

Measurement domain: The physical link between directly connected devices is a packet conservation domain. The measurement range

contains physical direct links, and TM chips and MAC chips on interfaces.

Object: All incoming and outgoing IP unicast flows of the measurement domain.

Unidirectional packet loss from device 1 to device 2 = C1_1 - C2_1

Unidirectional packet loss from device 2 to device 1 = C2_2 - C1_2

Note: The TM and MAC chips do not support iPCA. The measurement object is all packets. The measurement interval of TM and MAC

chips is not synchronized with that of micro engine. Therefore, the statistics are only used as a reference for fault location.

Device 1 Device 2

Micro

engine MAC

ENP card

TM

C1_1

C1_2 MAC

ENP card

Micro

engine TM

C2_1

C2_2

Ingress TLP

Egress TLP

Expected

measurement

range

Actual measurement range

36

Network-Level Packet Loss Measurement

Measurement domain: A domain consisting of non-agile

devices (including third-party devices) surrounded by agile

devices and the links between agile devices and the

measurement domain.

Object: All incoming and outgoing IP unicast flows of the

measurement domain. (The current version only supports

measurement on the service flows with known directions.)

Ingress TLP Egress TLP

Device A Device B

Device C Device E

C1 C2

C3

C4

C5

Number of lost packets from devices A/B to devices C/D/E =

(C1 + C2) - (C3 + C4 + C5)

Incoming

packets

Outgoing

packets

Measurement

domain

Note: The measurement object in this example is a

unidirectional service flow.

37

Service Path Hop-by-Hop Measurement

Service packet forwarding path detected by IP Tracert

Service flow characteristics: Service packets must have known source and destination IP addresses.

Path tracing: eSight searches for the source gateway according to the source IP address of the service flow. The source gateway performs

IP Tracert to the destination IP address of the service flow to trace the forwarding path between source and destination gateways. The

gateways deliver service flow characteristics to agile devices. The agile devices returns the service flow inbound interfaces (1, 3, 5, and 7)

and outbound interfaces (2, 4, 6, and 8) to eSight. The Layer 3 IP path of the service flow is determined.

Measurement method: Each agile device measures service packets on its inbound and outbound interfaces. Two neighboring interfaces can

calculate the number of lost service packets on each segment (ACH).

Constraint: The current version of iPCA only supports IP networks, but does not support MPLS VPN or GRE network. If load balancing

paths or active/standby paths are configured, the measurement result on only the path obtained by IP Tracert is displayed.

Terminal Terminal

S57 (source gateway) S57 (destination gateway) S127 S127

1 2 3 4 5 6 7 8

ACH1 ACH2 ACH3 ACH4 ACH5 ACH6 ACH7

eSight

38

Huawei Products Supporting iPCA

Model Version Remarks

eSight V200R005C00 NMS,SLA

S5720HI V200R006C00 Fixed-chassis Agile switch

S7700 V200R006C00 Modular Agile switch,ENP



The device must support iPCA and have an ENP card installed.

39

Huawei AtomEngine

40

Solution Architecture

BS

Enterprise Enterprise

Mobile Core

Meter

Controller

Meter

Meter Meter Meter

Manager

Performance Test

Hop-by-hop Hop-by-hop Hop-by-hop

Network E2E Test

1

2 3

1

2

3

Meter: Atom Meter

• Bypass network quality measurement • In-line real time flow quality measurement • Identify, Coloring, Statistics

Controller: SNC-A • Atom Meter discovery • Management agent

Manager: U2000+uTraffic/U2520 • Performance test visualization • Atom Meter management:

configuration , log, alarm

CSG ASG RSG

AtomEngine Solution

41

NE40E/CX600/ ME60/PTN6900

SNC-A Board

SNC – Smart Network Controller

One SNC board can manage 1K Atom Meters, maximum 8K per chassis

X3/X8/X16

42

Huawei AtomEngine

43

Innovative AtomEngine Technology

NP Inside

Copyright©2012 Huawei Technologies Co., Ltd. All Rights Reserved.

The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product

portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive

statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time

without notice.

HUAWEI ENTERPRISE ICT SOLUTIONS A BETTER WAY