Mastering Data Center QoS - …d2zmdbbm9feqrf.cloudfront.net/2013/eur/pdf/BRKRST-2509.pdf · Mastering Data Center QoS Lucien Avramov Technical Marketing Engineer ... 3G/s 2G/s Storage

© 2012 Cisco and/or its affiliates. All rights reserved. BRKRST-2509 Cisco Public

Mastering Data Center QoS Lucien Avramov

Technical Marketing Engineer

Data Center Switching Group

Nexus 6000 / 5000 / 3000 / 2000 / 1000v

[email protected]

BRKRST-2509

2

mailto:[email protected]


Data Center QoS Design Requirements

• The Data Center adds a number of new traffic types and requirements

‒ No Drop, IPC, Storage, Vmotion, …

• New Protocols and mechanisms

‒ 802.1Qbb, 802.1Qaz, ECN, …

What do we need to consider?

3

Spectrum of Design Evolution

Ultra Low Latency

• Queueing is designed out of the network whenever possible

• Nanoseconds matter

Warehouse Scale • ECN & Data Center TCP • Incast Loads on the server ports

slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8

blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8























Virtualized Data Center

• vMotion, iSCSI, FCoE, NAS, CIFS • Multi Tenant Applications • Voice & Video

HPC/GRID

• Predictable Low Latency • Bursty Traffic (workload

migration • iWARP & RoCE


Mastering Data Center QoS Lucien Avramov

Technical Marketing Engineer

Data Center Switching Group

Nexus 6000 / 5000 / 3000 / 2000 / 1000v

[email protected]

BRKRST-2509

4

mailto:[email protected]


Session Objective

At the end of the session, the participants should:

• Understand Data Center QoS Requirements and

Capabilities

• Understand QoS implementation on Nexus platforms

• Understand how to configure QoS on Nexus

WHY, WHEN and HOW of QoS

5


Mastering Data Center QoS

• Data Center QoS Requirements

• Nexus QoS Capabilities

• Nexus QoS Configuration

BRKRST-2509

6

1K Cisco Nexus

x86


Evolution of QoS Design

• Quality of Service is not just about protecting voice and video anymore

• Campus Specialization

‒ Desktop based Unified Communications

‒ Blended Wired & Wireless Access

• Data Center Specialization

‒ Compute and Storage Virtualization

‒ Cloud Computing

• Protocol convergence onto the fabric

‒ Storage – FCoE, iSCSI, NFS

‒ Inter-Process and compute communication (RCoE, vMotion, … )

Switching Evolution and Specialization

7

VMotion

FCoE


Evolution of QoS Design

• VoIP and Video are now mainstream technologies

• Ongoing evolution to the full spectrum of Unified Communications

• High Definition Executive Communication Application requires stringent

Service-Level Agreement (SLA)

‒ Reliable Service—High Availability Infrastructure

‒ Application Service Management—QoS

Where are we starting from

8


Media based Application Requirements

Voice vs. Video – At the Packet Level

9

20 msec

Voice Packets

Bytes

200

600

1000

Audio

Samples

1400

Time

200

600

1000

1400

33 msec

Video Packets

Video

Frame

Video

Frame

Video

Frame


Enterprise / Campus QoS Design Requirements

QoS for Voice and Video is implicit in current Networks – Medianet RFC 4594

10

Application

Class

Per-Hop

Behavior

Admission

Control

Queuing &

Dropping

Application

Examples

VoIP Telephony EF Required Priority Queue (PQ) Cisco IP Phones (G.711, G.729)

Broadcast Video CS5 Required (Optional) PQ Cisco IP Video Surveillance / Cisco Enterprise TV

Realtime Interactive CS4 Required (Optional) PQ Cisco TelePresence

Multimedia Conferencing AF4 Required BW Queue + DSCP WRED Cisco Unified Personal Communicator, WebEx

Multimedia Streaming AF3 Recommended BW Queue + DSCP WRED Cisco Digital Media System (VoDs)

Network Control CS6 BW Queue EIGRP, OSPF, BGP, HSRP, IKE

Call-Signaling CS3 BW Queue SCCP, SIP, H.323

Ops / Admin / Mgmt (OAM) CS2 BW Queue SNMP, SSH, Syslog

Transactional Data AF2 BW Queue + DSCP WRED ERP Apps, CRM Apps, Database Apps

Bulk Data AF1 BW Queue + DSCP WRED E-mail, FTP, Backup Apps, Content Distribution

Best Effort DF Default Queue + RED Default Class

Scavenger CS1 Min BW Queue (Deferential) YouTube, iTunes, BitTorent, Xbox Live



• A balanced fabric is a function of maximal throughput ‘and’ minimal loss => “Goodput”

• Application-level throughput (goodput):

‒ Given by the total bytes received from all senders divided by the finishing time of the last sender.

‒ “Understanding TCP Incast Throughput Collapse in Datacenter Networks”

“Goodput”

11

5 millisecond view Congestion Threshold exceeded

Data Center Design Goal: Optimizing the balance of end to

end fabric latency with the ability to absorb traffic peaks and prevent

any associated traffic loss



Trust Boundaries – What have we trusted?

12

Access-Edge Switches

Conditionally Trusted Endpoints

Example: IP Phone + PC

Secure Endpoint

Example: Software-protected PC

With centrally-administered QoS

markings

Unsecure Endpoint

Tru

st

Bo

un

dary

Tru

st

Bo

un

dary



• Data Centre architecture can be provide a new set of trust boundaries

• Virtual Switch extends the trust boundary into the memory space of the Hypervisor

• Converged and Virtualized Adapters provide for local classification, marking and queuing

What do we trust and where do classify and mark?

13

vPC

vPC

VM #4

VM #3

VM #2

N1KV – Classification, Marking & Queuing

COS Based Queuing in the

extended Fabric

Trust Boundary

CNA/A-FEX - Classification and Marking

N2K – CoS Marking

COS Based Queuing in the

extended Fabric

N5K – CoS/DSCP Marking, Queuing and Classification

N7K – CoS/DSCP Marking, Queuing and Classification

COS/DSCP Based Queuing in the

extended Fabric



Data Center Model : 4 and 8 Class Model

14

Time

Critical Data

Realtime

4-Class Model

Best Effort

Signaling / Control Call Signaling

Critical Data

Interactive Video

Voice

8-Class Model

Scavenger

Best Effort

Streaming Video

Network Control

Network Management

Realtime Interactive

Transactional Data

Multimedia Conferencing

Voice

12-Class Model

Bulk Data

Scavenger

Best Effort

Multimedia Streaming

Network Control

Broadcast Video

Call Signaling

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND_40/QoSIntro_40.html#wp61135

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND_40/QoSIntro_40.html



• We have non IP based traffic to consider again

‒ FCoE – Fibre Channel Over Ethernet

‒ RCoE – RDMA Over Ethernet

• DSCP is still marked but CoS will be required in Nexus Data Center designs

CoS or DSCP?

15

PCP/COS Network priority Acronym Traffic characteristics

1 0 (lowest) BK Background

0 1 BE Best Effort

2 2 EE Excellent Effort

3 3 CA Critical Applications

4 4 VI Video, < 100 ms latency

5 5 VO Voice, < 10 ms latency

6 6 IC Internetwork Control

IEEE 802.1Q-2005



• In this example of a Virtualized Multi-Tenant Data Center there is a

potential overlap/conflict with Voice/Video queuing assignments, e.g.

‒ COS 3 – FCoE ‘and’ Call Control

Where do we put the new traffic types?

16

Traffic Type Network Class COS Class, Property, BW Allocation

Infrastructure Control 6 Platinum, 10%

vMotion 4 Silver, 20%

Tenant

Gold, Transactional 5 Gold, 30%

Silver, Transactional 2 Bronze, 15%

Bronze, Transactional 1 Best effort, 10%

Storage FCOE 3 No Drop, 15%

NFS datastore 5 Silver

Non Classified Data 1 Best Effort






BRKRST-2509

17

1K Cisco Nexus

x86


Priority Flow Control

• Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1p

• When link is congested, CoS assigned to “no-drop” will be PAUSED

• Other traffic assigned to other CoS values will continue to transmit and rely on

upper layer protocols for retransmission

• Not only for FCoE traffic

FCoE Flow Control Mechanism – 802.1Qbb

18

Packet

R_

RD

Y

Fibre Channel

Transmit Queues Ethernet Link Receive Buffers

Eight

Virtual

Lanes

One One

Two Two

Three Three

Four Four

Five Five

Seven Seven

Eight Eight

Six Six

STOP PAUSE

B2B Credits


Enhanced Transmission Selection (ETS)

• Prevents a single traffic class of “hogging” all the bandwidth and starving other classes

• When a given load doesn’t fully utilize its allocated bandwidth, it is available to other classes

• Helps accommodate for classes of a “bursty” nature

Bandwidth Management – 802.1Qaz

19

Offered Traffic

t1 t2 t3

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic

3G/s

2G/s

3G/s Storage Traffic

3G/s

3G/s

LAN Traffic

4G/s

5G/s 3G/s

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s


Data Center Bridging Control Protocol

• Negotiates Ethernet capability’s : PFC, ETS, CoS values between DCB capable peer devices

• Simplifies Management : allows for configuration and distribution of parameters from one node to another

• Responsible for Logical Link Up/Down signaling of Ethernet and Fibre Channel

• DCBX is LLDP with new TLV fields

• The original pre-standard CIN (Cisco, Intel, Nuova) DCBX utilized additional TLV’s

• DCBX negotiation failures result in:

‒ per-priority-pause not enabled on CoS values

‒ vfc not coming up – when DCBX is being used in FCoE environment

DCBX Overview - 802.1Qaz

20

DCBX Switch

DCBX CNA

Adapter

dc11-5020-3# sh lldp dcbx interface eth 1/40

Local DCBXP Control information:

Operation version: 00 Max version: 00 Seq no: 7 Ack no: 0

Type/

Subtype Version En/Will/Adv Config

006/000 000 Y/N/Y 00

<snip>

https://www.cisco.com/en/US/netsol/ns783/index.html

https://www.cisco.com/en/US/netsol/ns783/index.html


Explicit Congestion Notification - TCP

ECN, an extension to TCP

21

ECN provides end-to-end congestion notification without dropping packets. Both the network infrastructure and the end hosts have to be capable of supporting ECN for it to function properly. ECN uses the two least significant bits in the Diffserv field in the IP header to encode four different values. During periods of congestion a router will mark the DSCP header in the packet indicating congestion (0x11) to the receiving host who should notify the source host to reduce its transmission rate.

N3K-1(config)# policy-map type network-qos traffic-priorities

N3K-1(config-pmap-nq)# class type network-qos class-gold

N3K-1(config-pmap-nq-c)# congestion-control random-detect ecn

The configuration for enabling ECN is very similar to the previous WRED example, so only the policy-map configuration with the ecn option is displayed for simplicity.

ECN Configuration

Diffserv field Values in the IP Header

0x00 – Non ECN-Capable Transport

0x10 - ECN Capable Transport (0)

0x01 – ECN Capable Transport (1)

0x11 – Congestion Encountered

WRED and ECN are always applied to the system policy

Notes: When configuring ECN ensure there are not any queuing policy-maps applied to the

interfaces. Only configure the queuing policy under the system policy.


ECN in Action! Incast Results

22

0

2000

4000

6000

8000

10000

12000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Go

od

pu

t in

Mb

ps

Server Incast #

TCP

ECN


DCTCP - Two Key Ideas

• React in proportion to the extent of congestion, not its presence.

‒ Reduces variance in sending rates, lowering queuing requirements.

• Mark based on instantaneous queue length.

‒ Fast feedback to better deal with bursts.

23

ECN Marks TCP DCTCP

1 0 1 1 1 1 0 1 1 1 Cut window by 50% Cut window by 40%

0 0 0 0 0 0 0 0 0 1 Cut window by 50% Cut window by 5%

Source: Data Center TCP (DCTCP), SIGCOMM 2010, New Dehli, India, August 31, 2010.

http://www.stanford.edu/~alizade/Site/DCTCP_files/DCTCP-talk.pptx



• Small Flows/Messaging

(Heart-beats, Keep-alive, delay sensitive application messaging)

• Small – Medium Incast

(Hadoop Shuffle, Scatter-Gather, Distributed Storage)

• Large Flows

(HDFS Insert, File Copy)

• Large Incast

(Hadoop Replication, Distributed Storage)

Diversity of Data Center Application Flows

24


Data Center QoS Design Buffering Considerations

• Storage

• Uplink Speed Mismatch

• Incast / Many to One conversations

When are Buffers Needed?

25 25

1GE

Access

10GE

10GE 10GE

10GE

1GE

Access


1 13 25 37 49 61 73 85 97 109

121

133

145

157

169

181

193

205

217

229

241

253

265

277

289

301

313

325

337

349

361

373

385

397

409

421

433

445

457

469

481

493

505

517

529

541

553

565

577

589

601

613

625

637

649

661

673

685

697

709

721

733

745

757

769

781

793

Job

Com

plet

ion

Cell U

sage

1G Buffer Used 10G Buffer Used 1G Map % 1G Reduce % 10G Map % 10G Reduce %


Buffer Amount – 1GE vs. 10GE Buffer Usage

26

10 GE Buffer

1 GE Buffer

Going to 10GE lowers the buffer utilization on switching layer


Remaining Logic (Forwarding, etc)

Tables (L2,L3,MC,etc)


Buffer Amount – The Buffer Bloat

Buffer

ASIC

27


Buffer Amount – The Switch Architecture

28

Egress Buffer

INGRESS

EGRESS

INGRESS

Ingress per port Buffer

Scheduler

Crossbar Egress per port Buffer

EGRESS

Shared

Memory Buffer

Scheduler

INGRESS

EGRESS

Crossbar

Scheduler

Cat6k

N5K N6K N7k

4900 N3K

… …


dc11-5020-4# sh queuing int eth 1/39

Interface Ethernet1/39 TX Queuing

qos-group sched-type oper-bandwidth

0 WRR 50

1 WRR 50

Interface Ethernet1/39 RX Queuing

qos-group 0

q-size: 243200, HW MTU: 1600 (1500 configured)

drop-type: drop, xon: 0, xoff: 1520

Statistics:

Pkts received over the port : 85257

Ucast pkts sent to the cross-bar : 930

Mcast pkts sent to the cross-bar : 84327

Ucast pkts received from the cross-bar : 249

Pkts sent to the port : 133878

Pkts discarded on ingress : 0

Per-priority-pause status : Rx (Inactive), Tx (Inactive)

<snip – other classes repeated>

Total Multicast crossbar statistics:

Mcast pkts received from the cross-bar : 283558

SFP SFP SFP SFP

Unified Crossbar Fabric

UPC

Egress (Tx) Queuing

Configuration

Packets Arriving on this port but

dropped from ingress queue due

to congestion on egress port

Nexus QoS

Mapping the Switch Architecture to ‘show queuing’ – Drops occur on Ingress

29






‒ Nexus Configuration Model: MQC

‒ Platform Configuration Examples

Nexus 7000

Nexus 5000 / 3000

Nexus 2000

Nexus 1000v

BRKRST-2509

30

1K Cisco Nexus

x86


Nexus QoS

• Nexus 1000v/3000/5000/6000/7000 supports a new set of QoS capabilities designed to provide per

system class based traffic control

‒ Lossless Ethernet—Priority Flow Control (IEEE 802.1Qbb)

‒ Traffic Protection—Bandwidth Management (IEEE 802.1Qaz)

‒ Configuration signaling to end points—DCBX (part of IEEE 802.1Qaz)

• These new capabilities are added to and managed by the common Cisco MQC (Modular QoS CLI) which

defines a three-step configuration model

‒ Define matching criteria via a class-map

‒ Associate action with each defined class via a policy-map

‒ Apply policy to entire system or an interface via a service-policy

• Nexus 1000v/3000/5000/6000/7000 leverage the MQC qos-group capabilities to identify and define traffic

in policy configuration

Capabilities and Configuration

31


Nexus QoS

Configuration Principles

32

• Class-Map

• Policy-Map

• Service-Policy 3 MQCs

• N1000v: 64 classes (8 pre-defined)

• N3K: 8 classes / Qos-groups (4 Multicast)

• N6K N5K: 6 classes

• N7K: 2 to 8 classes

Classes

• Type Network-QOS

• Type Queuing

• Type QOS

Policies


Type (CLI) Description Applied To…

QoS Packet Classification based on Layer 2/3/4 (Ingress) Interface or System

Network-QoS Packet Marking (CoS), Congestion Control WRED/ECN (Egress) System

Queuing Scheduling - Queuing Bandwidth % / Priority Queue (Egress) Interface or System

Nexus Software QoS Model

• QoS is enabled by default (NX-OS Default)

• Qos policy defines how the system classifies traffic, assigned to qos-groups

• Network QoS policy defines system policies, e.g. which COS values ALL ports treat as drop

versus no-drop

• Ingress queuing policy defines how ingress port buffers ingress traffic for ALL destinations over

fabric

• Egress queuing policy defines how egress port transmits traffic on wire

‒ Conceptually, controls how all ingress ports schedule traffic toward the egress port over fabric (by controlling

manner in which bandwidth availability reported to arbiter)

Overview

33


Classification ACL, CoS, DSCP, IP RTP,

Precedence, Protocol

Policy Sets qos-group to the system class

this traffic flow is mapped to

Classification System class matched by QoS Group

Policy MTU

Queue-Limit (5k)

Set CoS Mark 802.1p

Set DSCP (5500/3k)

ECN-WRED (3k)

Classification System class matched by qos-

group

Policy ETS Guaranteed scheduling deficit

weighted round robin (DWRR)

percentage

Priority

Strict priority scheduling – Only one

class can be configured for priority in

a given queuing policy

Shaping (7k)

type qos type network-qos type queuing

1 2 3

4 Apply service-policy


Nexus 7000 F2 / 6000 / 5500 / 3000

34



Configuration Steps

35

ACL app-2 qos-group=2 set cos 2 Buffer 82kb

Cos=2

STEP 1 Qos - Ingress STEP 2- Network-Qos STEP 3 – Queuing In/Egress

system qos

service-policy type qos input INGRESS_CLASS

service-policy type network-qos MARK_COS

service-policy type queuing output EGRESS_QUEUE

Class-map type qos class-app-1

Match access-group app-1

Class-map type qos class-app-2

Match access-group app-2

Policy-map type qos policy-qos

Clas type qos class-app-1

Set qos-group 1

Class type qos class-app-2

Set qos-group 2

Class-map type queuing class-app-

1

Match qos-group 1

Class-map type queuing class-app-

2

Match qos-group 2

Policy-map type queuing policy-

queue

Class type queuing class-default

Bandwidth percent 10

Class type queuing class-app-1


Class type queuing class-app-2


Class-map type network-qos class-

app-1

Match qos-group 1

Class-map type network-qos class-

app-2

Match qos-group 2

Policy-map type network-qos

policy-nq

Class type network-qos class-app-

1

Pause no-drop

MTU 9216

Class type network-qos class-app-

2

Set cos 2

Queue-limit 81920 bytes

STEP 4

Apply QoS – Global / per int.


Nexus QoS

• Define global queuing and scheduling parameters for all interfaces in switch

‒ Identify drop/no-drop classes, instantiate specific default queuing policies, etc.

• One network-QoS policy per system, applies to all ports in all VDCs

• Assumption is network-QoS policy defined/applied consistently network-wide

‒ Particularly for no-drop applications, end-to-end consistency mandatory

Consistent type ‘network-QoS’ Policies

36

Switch 1 Switch 2 Switch 3

Network QoS policies should be applied consistently on all switches network wide

Fabric

Ingress Module

Ingress Module

Ingress Module

Egress Module Fabric

Ingress Module

Ingress Module

Ingress Module

Egress Module Fabric

Ingress Module

Ingress Module

Ingress Module

Egress Module








Nexus 7000

Nexus 5000 / 3000

Nexus 2000

Nexus 1000v

BRKRST-2509

37

1K Cisco Nexus

x86


Modifying MTU

• MTU in network-QoS policy applies to all F1/F2 interfaces in absence of per-

port MTU configuration. User-configured per-port MTU overrides any MTU in

network-QoS policy (for that port)

• Per-port or network-QoS defined MTUs must be less than or equal to

configured system jumbomtu value

• L2 switchport MTU must be 1518 or the “system jumbomtu” value if MTU

configured per-port

• Example of per-port MTU (modifies MTU only on specified port):

F1 and F2 modules

N7K(config)# interface e3/1 N7K(config-if)# mtu 9216 N7K(config-if)#

38


Modifying MTU

• Example of network-QoS MTU (modifies MTU for specified class on all F1/F2

ports)

F1 and F2 modules

N7K# !Clone the 7E policy (cannot modify default policies) N7K# qos copy policy-map type network-qos default-nq-7e-policy prefix new- N7K# conf Enter configuration commands, one per line. End with CNTL/Z.

N7K(config)# !Modify the newly cloned policy-map N7K(config)# policy-map type network-qos new-nq-7e N7K(config-pmap-nqos)# !Modify the 7E drop class N7K(config-pmap-nqos)# class type network-qos c-nq-7e-drop N7K(config-pmap-nqos-c)# mtu 8000 N7K(config-pmap-nqos-c)# !Apply the new policy-map to the system qos target N7K(config-pmap-nqos-c)# system qos N7K(config-sys-qos)# service-policy type network-qos new-nq-7e N7K(config-sys-qos)#

39


What Is a Strict Priority Queue?

40

• In classic definition, SP queue gets complete, unrestricted access to all interface bandwidth and is serviced until empty

‒Can theoretically starve all other traffic classes

• Depending on hardware implementation, additional options for SP queue exist:

‒Multiple PQs with hierarchical relationship (e.g., level 1 vs. level 2)

‒Multiple PQs with bandwidth sharing according to DWRR weights

‒Optional SP queue shaping

M1 modules:

• SP queue adheres to classic SP queue definition

‒You cannot limit how much interface bandwidth traffic mapped to SP queue consumes

• Use care in mapping traffic to SP queue – SP traffic should be low volume

F1/F2 modules:

• Multiple SP queues can exist, depending on active network-QoS template

• SP queue(s) can be shaped to prevent complete starvation of other classes

‒Note that a shaped queue cannot exceed the shaped rate even if no congestion exists


Modifying Queuing Behavior

Shape the SP Queue on F1/F2 Modules

41

Clone a default egress “type queuing” policy-map

Creates a copy of a default egress queuing policy

Shape SP queue in new (cloned) “type queuing” policy Limit SP queue bandwidth consumption

Apply new “type queuing” policy to target interface(s)

Apply new queuing policy to F1/F2 interfaces

Important: applying new queuing policy takes effect immediately and is disruptive to any ports to which the policy is applied



Shape the SP Queue on F1/F2 Modules

42

• Example: Shape the SP queue to 2Gbps on an interface, using a queuing policy cloned from the default “8E” egress queuing policy

N7K# !Clone the 8E egress queuing policy

N7K# qos copy policy-map type queuing default-4q-8e-out-policy prefix new-

N7K# conf t

Enter configuration commands, one per line. End with CNTL/Z.

N7K(config)# !Modify new queuing policy

N7K(config)# policy-map type queuing new-4q-8e-out

N7K(config-pmap-que)# !Modify the SP queue

N7K(config-pmap-que)# class type queuing 1p3q1t-8e-out-pq1

N7K(config-pmap-c-que)# !Shape the queue to 20% (2G)

N7K(config-pmap-c-que)# shape percent 20

N7K(config-pmap-c-que)# !Apply the new policy to target interface

N7K(config-pmap-c-que)# int e 2/1

N7K(config-if)# service-policy type queuing output new-4q-8e-out

N7K(config-if)#



Make an Interface “Untrusted”

43

Create ingress “type queuing” policy-map to set COS to 0

Rewrites COS of all frames to 0 – only needed if ingress is 1Q trunk

Create “type qos” marking policy to set DSCP to 0

Rewrites DSCP of all IP packets to 0

Apply new policies to target interface(s)

Apply new policies to interfaces

Important: applying new queuing policy takes effect immediately and is disruptive to any ports to which the policy is applied


Modifying Queuing Behavior Make an Interface “Untrusted” – M Modules

44

• Example: Make a Layer 2 trunk port untrusted on M1 1G module N7K# conf t


N7K(config)# !create new queuing policy

N7K(config)# policy-map type queuing untrusted

N7K(config-pmap-que)# !use default ingress queuing class-map for target port type

N7K(config-pmap-que)# class type queuing 2q4t-in-q-default

N7K(config-pmap-c-que)# !Give q-default maximum buffer space

N7K(config-pmap-c-que)# queue-limit percent 100

N7K(config-pmap-c-que)# !change COS to 0 (can only 'set cos' in q-default)

N7K(config-pmap-c-que)# set cos 0

N7K(config-pmap-c-que)# !create new qos policy

N7K(config-pmap-c-que)# policy-map type qos untrusted

N7K(config-pmap-qos)# !use class-default to match everything

N7K(config-pmap-qos)# class class-default

N7K(config-pmap-c-qos)# !change DSCP of all packets to 0

N7K(config-pmap-c-qos)# set dscp 0

N7K(config-pmap-c-qos)# int e 3/1

N7K(config-if)# !tie the queuing & qos policies to the target interface(s)

N7K(config-if)# service-policy type queuing input untrusted

N7K(config-if)# service-policy type qos input untrusted

N7K(config-if)#

Note: for an access switchport, queuing policy not necessary since no COS received



Make an Interface “Untrusted” – F1/F2 Modules

45

• N7K# !Clone the default input queuing policy, or create a new one from scratch

N7K# qos copy policy-map type queuing default-4q-8e-in-policy prefix untrusted-

N7K# conf


N7K(config)# !Modify the cloned policy

N7K(config)# policy-map type queuing untrusted-4q-8e-in

N7K(config-pmap-que)# !For F1/F2, must specify all queues even for untrusted policy

N7K(config-pmap-que)# class type queuing 2q4t-8e-in-q1

N7K(config-pmap-c-que)# !Give q1 the minimum buffer space


N7K(config-pmap-c-que)# class type queuing 2q4t-8e-in-q-default

N7K(config-pmap-c-que)# !Give q-default maximum buffer space


N7K(config-pmap-c-que)# !Set COS 0 for all frames

N7K(config-pmap-c-que)# set cos 0

N7K(config-pmap-c-que)# policy-map type qos untrusted

N7K(config-pmap-qos)# !use class-default to match everything

N7K(config-pmap-qos)# class class-default

N7K(config-pmap-c-qos)# !change DSCP of all packets to 0

N7K(config-pmap-c-qos)# set dscp 0

N7K(config-pmap-c-qos)# int e1/1-32

N7K(config-if-range)# !tie the queuing & qos policies to the target interface(s)

N7K(config-if-range)# service-policy type queuing input untrusted-4q-8e-in

N7K(config-if-range)# service-policy type qos input untrusted

Note: for an access switchport, queuing policy not necessary since no COS received


Priority Flow Control – Nexus 7K

• No-Drop PFC w/ MTU 2K set for Fibre Channel

Operations Configuration – Switch Level

46

N7K-50(config)# system qos

N7K-50(config-sys-qos)# service-policy type network-qos default-nq-7e-policy

show policy-map system

Type network-qos policy-maps

=====================================

policy-map type network-qos default-nq-7e-policy

class type network-qos c-nq-7e-drop

match cos 0-2,4-7

congestion-control tail-drop

mtu 1500

class type network-qos c-nq-7e-ndrop-fcoe

match cos 3

match protocol fcoe

pause

mtu 2112 Template Drop CoS (Priority) NoDrop CoS (Priority)

default-nq-8e-policy 0,1,2,3,4,5,6,7 5,6,7 - -

default-nq-7e-policy 0,1,2,4,5,6,7 5,6,7 3 -

default-nq-6e-policy 0,1,2,5,6,7 5,6,7 3,4 4

default-nq-4e-policy 0,5,6,7 5,6,7 1,2,3,4 4

Policy Template choices

show class-map type network-qos c-nq-7e-ndrop-fcoe

Type network-qos class-maps

=============================================

class-map type network-qos match-any c-nq-7e-ndrop-fcoe

Description: 7E No-Drop FCoE CoS map

match cos 3

match protocol fcoe


Hierarchical Queuing Policies for ETS

• Enhanced Transmission Selection (ETS) provides priority group mappings and

bandwidth ratios

‒Controls hierarchical queuing policies for drop versus no-drop traffic classes

‒Defines bandwidth ratios advertised in DCBX for drop versus no-drop classes

• Only active when no-drop network-qos policy active (7E/6E/4E)

• Top-level policy-map defines overall queue-limit and bandwidth ratios for drop

versus no-drop classes

• Second-level policy-map defines priority, queue-limit, and bandwidth ratios for

individual drop and no-drop classes

F1 and F2 modules

47


Example of ETS Queuing Policy

F1 and F2 Modules

• Example using default queuing policies under 6E network-QoS template

• Top-level policy-map policy-map type queuing default-4q-6e-out-policy

class type queuing c-4q-6e-drop-out

service-policy type queuing default-4q-6e-drop-out-policy

bandwidth remaining percent 70

class type queuing c-4q-6e-ndrop-out

service-policy type queuing default-4q-6e-ndrop-out-policy


• Second-level policy-maps policy-map type queuing default-4q-6e-drop-out-policy

class type queuing 3p1q1t-6e-out-pq1

priority level 1

class type queuing 3p1q1t-6e-out-q-default


policy-map type queuing default-4q-6e-ndrop-out-policy


priority level 1


priority level 2

Defines overall

bandwidth ratio for drop

classes

Defines overall

bandwidth ratio for no-

drop classes

Defines priority and bandwidth

ratios for individual drop

classes

Defines priority and bandwidth

ratios for individual no-drop

classes

48








Nexus 7000

Nexus 6000 / 5000 / 3000

Nexus 2000

Nexus 1000v

BRKRST-2509

49

1K Cisco Nexus

x86


Configuration Templates

• MTU can be configured for each class of service (no interface level MTU)

• No fragmentation since Nexus 6000 / 5000 / 3000 is a L2 switch

• Cut-through, frames are truncated if they are larger than MTU

• Store-and-forward: frames are dropped if they are larger than MTU

• With L3 module (5000) or license (3000) L3 MTU at Routed Interface / SVI level

MTU – Nexus 6000 / 5000 / 2000 & 3000

50

Each CoS queue on the

Nexus 5000 supports a

unique MTU

class-map type qos iSCSI

match cos 2

class-map type queuing iSCSI

match qos-group 2

policy-map type qos iSCSI

class iSCSI

set qos-group 2

class-map type network-qos iSCSI

match qos-group 2

policy-map type network-qos iSCSI

class type network-qos iSCSI

mtu 9216

system qos

service-policy type qos input iSCSI

service-policy type network-qos iSCSI



• Nexus 6000 / 5000 / 3000 supports different MTU for each system class

• MTU is defined in network-qos policy-map

• L2: no interface level MTU support on Nexus 5000

• L3 MTU: at SVI / Routed port level

MTU – Nexus 6000 / 5000 / 3000

51

Policy-map type network-qos jumbo

Class type network-qos class-default

MTU 9216

System qos

Service-policy type network-qos jumbo

Interface ethernet 1/x

Mtu 9216

Each qos-group on the

Nexus 5000/3000 supports a

unique MTU



Configure Buffer Size

52


Configuration Templates – Nexus 5000 and 3000

Configure CoS Marking

53


Enhanced Transmission Selection

• Create classification rules first by defining and applying policy-map type qos

• Define and apply policy-map type queuing to configure strict priority and bandwidth sharing

Strict Priority and Bandwidth Management

54

pod3-5010-2(config)# class-map type queuing class-voice

pod3-5010-2(config-cmap-que)# match qos-group 2

pod3-5010-2(config-cmap-que)# class-map type queuing class-high


pod3-5010-2(config-cmap-que)# class-map type queuing class-low


pod3-5010-2(config-cmap-que)# exit

pod3-5010-2(config)# policy-map type queuing policy-BW

pod3-5010-2(config-pmap-que)# class type queuing class-voice

pod3-5010-2(config-pmap-c-que)# priority

pod3-5010-2(config-pmap-c-que)# class type queuing class-high

pod3-5010-2(config-pmap-c-que)# bandwidth percent 50

pod3-5010-2(config-pmap-c-que)# class type queuing class-low


pod3-5010-2(config-pmap-c-que)# class type queuing class-fcoe


pod3-5010-2(config-pmap-c-que)# class type queuing class-default


pod3-5010-2(config-pmap-c-que)# system qos

pod3-5010-2(config-sys-qos)# service-policy type queuing output policy-BW

FCoE Traffic given 20%

of the 10GE link

1Gig FC HBAs

1Gig Ethernet NICs

Traditional Server


Priority Flow Control – Nexus 5000/5500

• On Nexus 5000 once feature fcoe is configured, 2 classes are made by default

• class-fcoe is configured to be no-drop with an MTU of 2158

• Enabling the FCoE feature on Nexus 5548/96 does ‘not’ create

no-drop policies automatically as on Nexus 5010/20.

Operations Configuration – Switch Level

55

policy-map type qos default-in-policy

class type qos class-fcoe

set qos-group 1

class type qos class-default

set qos-group 0

policy-map type network-qos default-nq-policy

class type network-qos class-fcoe

pause no-drop

mtu 2158

system qos

service-policy type qos input fcoe-default-in-policy

service-policy type queuing input fcoe-default-in-policy

service-policy type queuing output fcoe-default-out-policy

service-policy type network-qos fcoe-default-nq-policy

FCoE DCB Switch

DCB CNA Adapter


Nexus 5000/5500 QoS

• Tuning of the lossless queues to support a variety of use cases

• Extended switch to switch no drop traffic lanes

‒ Support for 3km with Nexus 5500 and 5500

Priority Flow Control and No-Drop Queues

56

Configs for 3000m no-

drop class Buffer size

Pause Threshold

(XOFF)

Resume Threshold

(XON)

N5020 143680 bytes 58860 bytes 38400 bytes

N5548 152000 bytes 103360 bytes 83520 bytes

Support for 3 km no drop switch to switch links Inter Building DCB FCoE links

5548-FCoE(config)# policy-map type network-qos 3km-FCoE

5548-FCoE(config-pmap-nq)# class type network-qos 3km-FCoE

5548-FCoE(config-pmap-nq-c)# pause no-drop buffer-size 152000 pause-threshold 103360

resume-threshold 83520

Gen 2 UPC

Unified Crossbar Fabric

Gen 2 UPC


Configuring QoS on the Nexus 5500

Check System Classes

57








Nexus 7000

Nexus 5000 / 3000

Nexus 2000

Nexus 1000v

BRKRST-2509

58

1K Cisco Nexus

x86


Nexus 2248TP-E

• Speed mismatch between 10G NAS and 1G server requires QoS tuning

• Nexus 2248TP-E utilizes a 32MB shared buffer to handle larger traffic bursts

• Hadoop, NAS, AVID are examples of bursty applications

• You can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host)

• You can use a lower queue limit value on the Fabric Extender to prevent one blocked receiver from affecting traffic that is sent to other non-congested receivers ("head-of-line blocking”)

32MB Shared Buffer

59

N5548-L3(config-fex)# hardware N2248TPE queue-limit 4000000 rx

N5548-L3(config-fex)# hardware N2248TPE queue-limit 4000000 tx

N5548-L3(config)#interface e110/1/1

N5548-L3(config-if)# hardware N2348TP queue-limit 4096000 tx

VM #4

VM #3

VM #2

NAS iSCSI

10G Attached Source (NAS Array)

1G Attached Server

10

G

NFS

Tune 2248TP-E to support a extremely large

burst (Hadoop, AVID, …)


Nexus 2248TP-E Counters

60

N5596-L3-2(config-if)# sh queuing interface e110/1/1

Ethernet110/1/1 queuing information:

Input buffer allocation:

Qos-group: 0

frh: 2

drop-type: drop

cos: 0 1 2 3 4 5 6

xon xoff buffer-size

---------+---------+-----------

0 0 65536

Queueing:

queue qos-group cos priority bandwidth mtu

--------+------------+--------------+---------+---------+----

2 0 0 1 2 3 4 5 6 WRR 100 9728

Queue limit: 2097152 bytes

Queue Statistics:

---+----------------+-----------+------------+----------+------------+-----

Que|Received / |Tail Drop |No Buffer |MAC Error |Multicast |Queue

No |Transmitted | | | |Tail Drop |Depth

---+----------------+-----------+------------+----------+------------+-----

2rx| 5863073| 0| 0| 0| - | 0

2tx| 426378558047| 28490502| 0| 0| 0| 0

---+----------------+-----------+------------+----------+------------+-----

Ingress queue limit(Configurable)

Egress queue limit(Configurable)

Egress queues: CoS to queue mapping Bandwidth allocation MTU

Per port per queue counters

Drop due to oversubscription








Nexus 7000

Nexus 5000 / 3000

Nexus 2000

Nexus 1000v

BRKRST-2509

61

1K Cisco Nexus

x86


Cisco Nexus 1000V QoS Overview

• The Cisco Nexus 1000V offers a rich, data-center-focused QoS feature

set.

• Adheres to the Cisco Modular QoS CLI configuration model:

‒ Class maps

‒ Policy maps

‒ Service policies

Nexus 1000V

62


Traffic Classification

Traffic Marking

Policing

Classification Criteria:

• CoS

• IP precedence

• DSCP

• Layer 2 to Layer 4 parameters

• ACL

• QoS group

• Discard class

Classification

63


Marking

64

Marking allowed:

• CoS

• IP precedence

• DSCP

• QoS group

• Discard class


Traffic Marking

Policing


Policing

65

Policing Conditions:

• Conforms to rate limits

• Exceeds rate limit maximum

• Violates rate limit

Policing Types:

• Single rate (CIR)

• Dual rate (CIR and PIR)

• Color aware Traffic Classification

Traffic Marking

Policing


Using an MQC Object

• Ingress QoS policies per interface:

‒ One type QoS

‒ One type queuing

• Egress QoS policies per interface:

• One type QoS

• One type queuing

66

Ethernet 22

Nexus 1000V



• Organization of traffic into classes

• Match criteria are used to achieve classification

• One class can have multiple class criteria

• “And / Or” rules can be used for classification

67


Configuring a Class Map

• Example:

n1000v# config t

n1000v(config)# class-map [type qos] [match-any | match-all] class_map_name

n1000v(config-cmap-qos)# match [match definition]

n1000v(config-cmap-qos)# show class-map name

68


Configuring Type QoS Class Maps

n1000v(config)# class-map ?

WORD Class-map name (Max Size 40)

match-all Logical-AND all match statements under this classmap

match-any Logical-OR all match statements under this classmap

type Specify the type of this class-map

n1000v(config)# class-map type qos ?


match-all Logical-AND all match statements under this classmap

match-any Logical-OR all match statements under this classmap

n1000v(config)# class-map type qos m

match-all match-any

n1000v(config)# class-map type qos match-any ?


n1000v(config)# class-map type qos match-any DSCP-CLUB

n1000v(config-cmap-qos)# match dscp af11



n1000v(config-cmap-qos)# exit

n1000v(config)# show class-map DSCP-CLUB

Type qos class-maps

====================

class-map type qos match-any DSCP-CLUB

match dscp 10,12,38

69


Configuring Type QoS Class Maps (Cont.)

n1000v(config)# class-map type qos L2PROTOCOL

n1000v(config-cmap-qos)# match protocol arp

n1000v(config-cmap-qos)# match protocol cdp

n1000v(config-cmap-qos)# exit

n1000v(config)# show class-map L2PROTOCOL

Type qos class-maps

====================

class-map type qos match-all L2PROTOCOL

match protocol arp

match protocol cdp

n1000v(config)# class-map type qos match-any PACKET-SIZE

n1000v(config-cmap-qos)# match packet ?

length Length of IP datagram

n1000v(config-cmap-qos)# match packet length 1500

n1000v(config)# show class-map PACKET-SIZE

Type qos class-map

====================

class-map type qos match-any PACKET-SIZE

match packet length 1500

70


QoS Policies

QoS policy map Class of traffic

1 class map

Class of traffic 2

class map

Class of traffic N

class map

Class of traffic unmatched class default

Policy actions • Marking • Policing




71


Configuring QoS Marking Policies

• Example:

n1000v# config t

n1000v(config)# policy-map [type qos] [match-first] policy-map-name

n1000v(config-pmap-qos)# class [type qos] {class_map_name | class-

default}

n1000v(config-pmap-c-qos)# set dscp | precedence | cos | qos-group |

discard-class dscp-value

n1000v(config-pmap-c-qos)# show policy-map policy-map-name

72


Applying Ingress and Egress Policies

• Once the policy map has been configured, apply to an interface

as follows:

n1000v# config t

n1000v(config)# interface x/x

n1000v(config-if)# service-policy [input | output] policy-map-name

n1000v(config-if)# show service-policy service-policy-name

73


Policing

• Monitoring of data for a particular class

• Three conditions are monitored:

‒ Conform

‒ Exceed

‒ Violate

• Single action can be applied per condition

• Single- or dual-rate policers can be defined

74


Configuring a Policer

• Example:

n1000v# config t

n1000v(config)# policy-map policy1

n1000v(config-pmap-qos)# class class-default

n1000v(config-pmap-c-qos)# police cir 256000 pir 256000 conform transmit

exceed set dscp dscp table cir-markdown-map violate drop

n1000v(config-pmap-c-qos)# show policy-map policy1

75


QoS Statistics

• Each parameter has its own statistic:

‒ Policy

‒ Class action

‒ Match criteria

• Statistics are maintained per interface

• Statistics are enabled by default

• Can be globally disabled with the no qos statistics command

76


QoS Statistics Configuration Example

n1000v# config t

n1000v(config)#

n1000v(config)# qos statistics

n1000v(config)# show policy-map interface

77


Class-Based Weighted Fair Queuing on Nexus 1000V

• Provide bandwidth

guarantee for up to 64 total

queues on uplinks

• User defined Queues

• 8 Predefined traffic classes

‒ For VMware and N1KV

protocol traffic

• Queuing configured via

MQC

20%

30%

15%

5%

15%

15%

vMotion

VM_Platinum

VM_Gold

Default

ESX_Mgmt

N1K_Control,N1K_Packet

VM VM VM VMK NIC vMotion

78


Class-Based Weighted Fair Queuing on Nexus 1000V

• Configurable up to 56 custom queuing classes of VM, vApp data and other traffic – Validated 18

• Each queue can have a queue limit (# of packets)

‒ In 4.2(1)SV1(4a) limit is removed

• Queuing is done per physical uplink outbound

• 8 predefined protocol classes:

‒ vMotion

‒ FT-Logging

‒ iSCSI

‒ NFS

‒ ESX Management

‒ N1K Control

‒ N1K Packet

‒ N1K Management

79


Summary

• Data Center QoS requires characterization beyond voice and video.

• New capabilities: PFC, ETS , DCBX

• Platform consistency: MQC

• Platform dependencies : where PFC, PQ, Queue structure

• Different type of congestions / traffic flows

• More to QoS than Buffer Tuning: Application and transport tuning

• How to configure QoS on Nexus switches

80


Recommended Readings

QoS and The Data Center Fabric

81

© 2012 Cisco and/or its affiliates. All rights reserved. TECDCT-2001 Cisco Public

Please be aware of what other sessions you are attending (avoid

duplication and make the best use of your time at Cisco Live)

82

BRKDCT-2081 Cisco FabricPath Technology and Design

TECDCT-2145 Cisco FabricPath

BRKDCT-2048 Deploying Virtual Port Channel in NXOS

BRKDCT-2214 Ultra Low Latency Data Center Design - End-to-end design approach

BRKARC-3470 Cisco Nexus 7000 Hardware Architecture

BRKARC-3452 Cisco Nexus 5000/5500 and 2000 Switch Architecture

BRKARC-3453 Nexus 6000 - Architecture of the next-generation Switch for the datacenter

BRKCOM-2001 UCS Deep Dive

BRKCOM-2002 UCS Supported Storage Architectures and Best Practices with Storage

TECDCT-3297 Operating and Deploying NX-OS

BRKDCT-3060 Deployment challenges with Interconnecting Data Centers

LTRDCT-1142 FabricPath Deployment in the Data Center Lab

. . . (and many more)

Related Sessions

82 82

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public © 2012 Cisco and/or its affiliates. All rights reserved. BRKAPP-2027 Cisco Public 83

Documents

Mastering Data Center QoS - …d2zmdbbm9feqrf.cloudfront.net/2013/eur/pdf/BRKRST-2509.pdf · Mastering Data Center QoS Lucien Avramov Technical Marketing Engineer ... 3G/s 2G/s Storage