Upload
tranliem
View
259
Download
5
Embed Size (px)
Citation preview
Cisco Nexus 7000 / 7700 Switch Architecture
BRKARC-3470
Tim Stevenson Distinguished Engineer, Technical Marketing
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 3
Session Abstract
This session presents an in-depth study of the architecture of the latest generation of Nexus 7000 and Nexus 7700 data center switches. Topics include supervisors, fabrics, I/O modules, forwarding engines, and physical design elements, as well as a discussion of key hardware-enabled features that combine to implement high-performance data center network services.
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 4
Session Goal
• To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions
• This session will examine the Nexus 7700 system, as well as the latest additions to the Nexus 7000
• This session will not examine NX-OS software architecture or other Nexus platform architectures
4
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 5
What Is Nexus 7000?
Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection
Nexus 7000 designed for general-purpose Data Center deployments, focused on 10G density plus 40G/100G
I/O Modules
Supervisor Engines
Fabrics
Chassis
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 6
What Is Nexus 7700?
Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection
Nexus 7700 designed for SP and MSDC Data Center deployments, focused on high-density 40G/100G
I/O Modules
Supervisor Engine
Fabrics Chassis
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 7
Nexus 7000 General purpose DC switching w/10/40/100G
Nexus 7700 Targeted at Dense 40G/100G deployments
Com
mo
n F
oundatio
n
• Same release vehicles, versioning, feature-sets
• Common configuration model
• Common operational model
• Common fabric ASICs (Fab2) and architecture
• Same central arbitration model
• Same VOQ/QoS model
• Identical forwarding ASICs (F2E, F3)
• Consistent hardware feature sets
• Parallel evolution of hardware capability/scale
Nexus 7000 / Nexus 7700 – Common Foundation
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 8
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 9
Nexus 7000 Chassis Family
Front Rear
21RU
N7K-C7010
25RU
Front Rear N7K-C7018
Front Rear N7K-C7009
14RU
NX-OS 4.1(2) and later
NX-OS 5.2(1) and later
Nexus 7010 Nexus 7018
Nexus 7009
Front N7K-C7004
7RU
NX-OS 6.1(2) and later
Rear
Nexus 7004
Front
Back
Side Side
Side Side Side
Back
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 10
Nexus 7700 Chassis Family
Front Rear
26RU
N77-C7718
Nexus 7718
Front Rear
14RU
N77-C7710
Nexus 7710
Front Rear
9RU
N77-C7706
Nexus 7706
NX-OS 6.2(6) and later
NX-OS 6.2(2) and later
NX-OS 6.2(2) and later
Front
Back
Front
Back
Front
Back
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 11
Key Chassis Components
Nexus 7000
• Common components:
– Supervisor engines
– I/O modules
– Power supplies (except 7004)
• Chassis-specific components:
– Fabric modules
– Fan trays
Nexus 7700
• Common components:
– Supervisor engines
– I/O modules
– Power supplies
• Chassis-specific components:
– Fabric modules
– Fan trays
Common hardware components between Nexus 7000 and Nexus 7700: NONE
No interchangeable hardware components between Nexus 7000 and Nexus 7700
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 12
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 13
Supervisor Engine 2 / 2E
• Next generation supervisors providing control plane and management functions
• Connects to fabric via 1G inband interface
• Interfaces with I/O modules via 1G switched EOBC
• Second-generation dedicated central arbiter ASIC
– Controls access to fabric bandwidth via dedicated arbitration path to I/O modules
Console Port Management
Ethernet
N7K-SUP2/N7K-SUP2E
USB Host
Ports
ID and Status
LEDs
Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700)
Base performance High performance
One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM
USB Log
Flash
USB Expansion
Flash
N77-SUP2E
ID and Status
LEDs
Console Port Management
Ethernet
USB Expansion
Flash
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 14
Nexus 7000 / 7700 I/O Module Families
M1 1G and 10G
M2 10G / 40G / 100G
F1 10G F2 10G F2E 10G F3 40G
F2E 10G F3 10G / 40G / 100G
F3 closes the
F/M feature gap!
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 15
Nexus 7000 M2 I/O Modules
• 10G / 40G / 100G M2 I/O modules
• Share common hardware architecture
• Two integrated forwarding engines (120Mpps)
– Support for “XL” forwarding tables (licensed)
• Distributed L3 multicast replication
• 802.1AE LinkSec on all ports
• Supports Nexus 2000 (FEX) connections
N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
N7K-M224XP-23L
Supported in NX-OS release 6.1(1) and later
N7K-M206FQ-23L
N7K-M202CF-22L
Module Port Density Optics Bandwidth
M2 10G 24 x 10G (plus Nexus 2000 FEX support) SFP+ 240G
M2 40G 6 x 40G (or up to 24 x 10G via breakout) QSFP+ 240G
M2 100G 2 x 100G CFP 200G
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 16
Nexus 7000 M2 I/O Module Architecture N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
LinkSec +
12 X 10G MAC -or-
3 X 40G MAC -or-
1 X 100G MAC
Forwarding
Engine
VOQs
Fabric 2 ASIC
To Fabric Modules
Replication
Engine
Replication
Engine
Front Panel Ports
LC
CPU
EOBC
VOQs
LinkSec +
12 X 10G MAC -or-
3 X 40G MAC -or-
1 X 100G MAC
Forwarding
Engine
VOQs
Replication
Engine
Replication
Engine
VOQs
To Central Arbiters
Arbitration
Aggregator …
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 17
Reference: ASIC Functions – M2 Modules
• LinkSec + MAC – Provides port ASIC functions, including buffering/queuing, and performs 802.1ae encryption/decryption for front-panel ports
• Replication Engine – Bridge between front panel port, forwarding engine, and fabric; performs multicast and SPAN replication
• Forwarding Engine – Performs all Layer 2, Layer 3, and Layer 4 forwarding decisions and policy enforcement
• VOQs – Interface to central arbiter and local crossbar fabric, implements Virtual Output Queuing
• Arbitration Aggregator – Muxes arbitration requests from VOQs before sending to central arbiter on Supervisor Engine
• Fabric 2 – Local fabric that provides first/third stage of three-stage crossbar
• (LC CPU – Linecard CPU, runs module-specific NX-OS processes and interfaces with Supervisor Engine over EOBC)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 18
Nexus 7000 / 7700 F2E I/O Modules
• 48-port 1G/10G with SFP/SFP+ transceivers
• 480G full-duplex fabric connectivity
• System-on-chip (SoC) forwarding engine design
– 12 independent SoC ASICs
• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS)
• Interoperability with M1/M2, in Layer 2 mode on Nexus 7000
– Proxy routing for inter-VLAN/L3 traffic
• LinkSec support*
– Last 8 ports (SFP+)
– All 48 ports (Copper)
• Supports Nexus 2000 (FEX) connections
N7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E
7000: Supported in NX-OS release 6.1(2) and later
7700: Supported in NX-OS release 6.2(2) and later
N7K-F248XP-25E N7K-F248XT-25E
* Roadmap item
N77-F248XP-23E
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 19
Nexus 7000 F2E Module Architecture N7K-F248XP-25E / N7K-F248XT-25E
4 X 10G
SoC
Front Panel Ports
To Fabric Modules
Fabric 2
2 4
LC
CPU
EOBC To Central Arbiters
Arbitration
Aggregator …
4 X 10G
SoC
6 8
4 X 10G
SoC
10 12
4 X 10G
SoC
14 16
4 X 10G
SoC
18 20
4 X 10G
SoC
22 24
4 X 10G
SoC
26 28
4 X 10G
SoC
30 32
4 X 10G
SoC
34 36
4 X 10G
SoC
38 40
4 X 10G
SoC
42 44
4 X 10G
SoC
46 48
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
LinkSec-capable (F2E fiber)
LinkSec-capable (F2E copper)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 20
Nexus 7700 F2E Module Architecture N77-F248XP-23E
4 X 10G
SoC
Front Panel Ports
To Fabric Modules
Fabric 2
2 4
LC
CPU
EOBC To Central Arbiters
Arbitration
Aggregator …
4 X 10G
SoC
6 8
4 X 10G
SoC
10 12
4 X 10G
SoC
14 16
4 X 10G
SoC
18 20
4 X 10G
SoC
22 24
4 X 10G
SoC
26 28
4 X 10G
SoC
30 32
4 X 10G
SoC
34 36
4 X 10G
SoC
38 40
4 X 10G
SoC
42 44
4 X 10G
SoC
46 48
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
LinkSec-capable
Fabric 2
To Fabric Modules
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 21
Reference: ASIC Functions – F2E Modules
• 4 X 10G SoC – Four-port 10G system-on-chip; provides Port ASIC, Replication Engine, Forwarding Engine, and VOQ functions
• Arbitration Aggregator – Muxes arbitration requests from SoCs before sending to central arbiter on Supervisor Engine
• Fabric 2 – Local fabric that provides first/third stage of three-stage crossbar
• (LC CPU – Linecard CPU, runs module-specific NX-OS processes and interfaces with Supervisor Engine over EOBC)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 22
Nexus 7000 F3 40G Module
• 12-port 40G QSFP+ module
• 480G full-duplex fabric connectivity
• SoC forwarding engine design
– 6 independent SoC ASICs
• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features
• Fabric Services Accelerator (FSA) CPU
• Breakout cable support
• Requires Supervisor Engine 2 / 2E
N7K-F312FQ-25
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 23
Nexus 7000 F3 12-Port 40G Module Architecture
1
Front Panel Ports (QSFP+)
To Fabric Modules
FSA
CPU
EOBC To Central Arbiters
Arbitration
Aggregator
2 X 40G
SoC 1
2 X 40G
SoC 2
2 X 40G
SoC 3
2 X 40G
SoC 4
2 X 40G
SoC 5
2 X 40G
SoC 6
Fabric ASIC
LC Inband
2 3 4 5 6 7 8 9 10 11 12
… x 6
to FSA
CPU to ARB
x 6
1G switch
x 6
…
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 24
FSA CPU
Fabric Services Accelerator (FSA)
• High-performance module CPU with on-board acceleration engines
– 6Gbps inband connectivity from SOCs to FSA
– Multi-Mpps packet processing
– 2 X 2GB dedicated DRAM
• Performance/scale boost for distributed fabric services, including BFD and sampled NetFlow (roadmap)
• Other potential applications include distributed ARP/ping processing, data plane packet analysis (wireshark), network probing, etc.
6 x 1Gbps
Module Inband
I/O
2GB DRAM
Dual-Core LC CPU
Acceleration Engines
2GB DRAM
EOBC
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 25
Nexus 7700 F3 48-Port 1G/10G Module
• 48-port 1G/10G with SFP/SFP+ transceivers
• 480G full-duplex fabric connectivity
• SoC-based forwarding engine design
– 6 independent SoC ASICs
• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features
• Fabric Services Accelerator (FSA) CPU
• LinkSec support (last 8 ports)*
• Supports Nexus 2000 (FEX) connections N77-F348XP-23
* Roadmap item
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 26
8 X 10G
SoC 1
Nexus 7700 F3 48-Port 1G/10G Module Architecture
To Fabric Modules To Central Arbiters
Arbitration
Aggregator
8 X 10G
SoC 2
8 X 10G
SoC 3
8 X 10G
SoC 4
8 X 10G
SoC 5
8 X 10G
SoC 6
Fabric ASIC Fabric ASIC … x 6
1
Front Panel Ports (SFP/SFP+)
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
LinkSec-capable
to FSA
CPU to ARB
FSA
CPU
EOBC
LC Inband
x 6
1G switch
x 6
…
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 27
Nexus 7700 F3 40G and 100G Modules
• 24-port 40G QSFP+ module / 12-port 100G CPAK module
• 960G/1.2T full-duplex fabric connectivity
• SoC forwarding engine design
– 12 independent SoC ASICs
• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features
• Fabric Services Accelerator (FSA) CPU
• 40G breakout cable support
N77-F324FQ-25
N77-F312CK-26
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 28
2 X 40G
SoC 1
Nexus 7700 F3 24-Port 40G Module Architecture
1
Front Panel Ports (QSFP+)
To Fabric Modules
FSA
CPU
EOBC To Central Arbiters
Arbitration
Aggregator
2 X 40G
SoC 2
2 X 40G
SoC 3
2 X 40G
SoC 4
2 X 40G
SoC 5
2 X 40G
SoC 6
2 X 40G
SoC 7
2 X 40G
SoC 8
2 X 40G
SoC 9
2 X 40G
SoC 10
2 X 40G
SoC 11
2 X 40G
SoC 12
Fabric ASIC Fabric ASIC
LC Inband
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
1G switch
…
… x 12
to FSA
CPU to ARB
x 12
x 6
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 29
Nexus 7700 F3 12-Port 100G Module Architecture
Front Panel Ports (CPAK)
To Fabric Modules To Central Arbiters
Arbitration
Aggregator
1 X 100G
SoC 2
2
1 X 100G
SoC 3
3
1 X 100G
SoC 4
4
1 X 100G
SoC 5
5
1 X 100G
SoC 6
6
1 X 100G
SoC 7
1 X 100G
SoC 8
1 X 100G
SoC 9
1 X 100G
SoC 10
1 X 100G
SoC 11
Fabric ASIC Fabric ASIC
7 8 9 10 11
1 X 100G
SoC 12
12
1 X 100G
SoC 1
1
FSA
CPU
EOBC
LC Inband
1G switch
…
… x 12
to FSA
CPU to ARB
x 12
x 6
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 30
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 31
M-Series Forwarding Engine Hardware
• Two hardware forwarding engines integrated on every M2 I/O module
• 120Mpps (60Mpps per forwarding engine) Layer 2 bridging with hardware MAC learning
• 120 Mpps (60Mpps per forwarding engine) Layer 3 IPv4
• 60Mpps (30Mpps per forwarding engine) Layer 3 IPv6 unicast
• Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir)
• MPLS/VPLS/EoMPLS
• OTV
• RACL/VACL/PACL
• QoS remarking and policing policies
• Policy-based routing (PBR)
• Unicast RPF check and IP source guard
• IGMP snooping
• Ingress and egress NetFlow (full and sampled)
Hardware Table M-Series Modules
without Scale License
M-Series Modules with
Scale License
MAC Address Table 128K 128K
FIB TCAM 128K IPv4 / 64K IPv6 900K IPv4 / 350K IPv6
Classification TCAM (ACL/QoS) 64K 128K
NetFlow Table 1M 1M
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 32
From I/O Module
Replication Engines
To I/O Module
Replication Engines
M-Series Forwarding Engine Architecture
L2 Engine
Ingress Parser
MAC
Table L2 Lookup (pre-L3)
L2 Lookup (post-L3)
Final Results
L3 Engine
Classification
(ACL/QoS)
NetFlow
Layer 3 FIB
Policing
FIB TCAM/
ADJ
CL TCAM
FE Daughter Card
Ingress lookup pipeline
Egress lookup
pipeline
Egress NetFlow collection
Ingress MAC table lookups
Port-channel hash result
Ingress IGMP snooping
lookups
FIB TCAM and adjacency table
lookups for Layer 3 forwarding
ECMP hashing
Multicast RPF check
Ingress policing
Egress MAC lookups
Egress IGMP snooping
lookups
PKT
HDR
Egress ACL/QoS classification
Ingress NetFlow collection
Ingress ACL/QoS classification
Egress policing
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 33
F2E Forwarding Engine Hardware
• Each SoC forwarding engine services 4 front-panel 10G ports (12 SoCs per module)
• 60Mpps per SoC Layer 2 bridging with hardware MAC learning
• 60Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast
• Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)
• RACL/VACL/PACL
• QoS remarking and policing policies
• Policy-based routing (PBR)
• Unicast RPF check and IP source guard
• IGMP snooping
• Ingress sampled NetFlow
• FabricPath forwarding
• FCoE (with Sup2 / Sup2E)
– Roadmap on Nexus 7700
Hardware Table Per F2E SoC Per F2E Module
MAC Address Table 16K 192K*
FIB TCAM 32K IPv4/16K IPv6 32K IPv4/16K IPv6
Classification TCAM (ACL/QoS) 16K 192K*
* Assumes specific configuration to scale SoC resources
* Roadmap item
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 34
F3 Forwarding Engine Hardware
• Each SoC forwarding engine services:
– 8 front-panel 10G ports
– 2 front-panel 40G ports
– 1 front-panel 100G port
• 148Mpps per SoC Layer 2 bridging with hardware MAC learning
• 148Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast
• Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)
• RACL/VACL/PACL
• QoS remarking and policing policies
• Policy-based routing (PBR)
• Unicast RPF check and IP source guard
• IGMP snooping
• Ingress/egress* sampled NetFlow
• FabricPath forwarding
• Overlay Transport Virtualization (OTV)
• MPLS/VPLS/EoMPLS, LISP, VXLAN, GRE, FCoE*
Hardware Table Per F3 SoC Per F3 Module
MAC Address Table 64K 384K/768K**
FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6
Classification TCAM (ACL/QoS) 16K 96K/192K**
** Assumes specific configuration to scale SoC resources
* Roadmap items
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 35
Front-panel
F3 Forwarding Engine
Decision Engine
Layer 3 Lookups
QoS / ACL
Ingress Parser
MAC
Table
FIB/ADJ
CL
L2 Lookup (post-L3)
To/From Central
Arbiter To Fabric From Fabric
Ingress
Buffer
L2 Lookup (pre-L3)
Egress Parser
F3 SoC
Ingress and egress
forwarding decisions
(L2/L3 lookups,
ACL/QoS, features etc.)
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
Forwarding
tables
1G / 10G / 40G / 100G
1G / 10G / 40G / 100G
capable interface MAC
Egress
Buffer Egress fabric
receive buffer
HDR
PKT HDR
PKT
PKT HDR Virtual
Queuing
Ingress buffer
memory
VOQ
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 36
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 37
Crossbar Switch Fabric Modules
• Provide interconnection of I/O modules
• Each installed fabric increases available per-payload slot bandwidth
• Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC
• Different I/O modules leverage different amount of available fabric bandwidth
• Access to fabric bandwidth controlled using QoS-aware central arbitration with VOQ
N7K-C7018-FAB-2
N7K-C7010-FAB-2
N7K-C7009-FAB-2
Fabric Module Supported Chassis Per-fabric module
bandwidth
Max fabric
modules
Total bandwidth per
slot
Nexus 7000 Fabric 2 7009 / 7010 / 7018 110Gbps per slot 5 550Gbps per slot
Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot
N77-C7718-FAB-2
N77-C7710-FAB-2
N77-C7706-FAB-2
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 38
Egress
Module
Ingress
Module
110G
(2 x 55G)
Ingress Module Egress Module
Multistage Crossbar
Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric
• Stages 1 and 3 on I/O modules
• Stage 2 on fabric modules
1st stage
2nd stage
3rd stage Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC
Fabric Modules
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
1
Fabric
ASIC 2 3 4 5
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC 6
Fabric
ASIC
1.32T
1st stage
3rd stage
550G
110G
(2 x 55G)
1 Fabric
ASIC
2 3 4 5 Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric Modules
Nexus 7000 Nexus 7700
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 39
110Gbps 220Gbps 330Gbps 440Gbps 550Gbps
Local Fabric 2
(480G)
Local Fabric 2
(240G)
I/O Module Capacity – Nexus 7000
One fabric:
• Any port can pass traffic to any other port in VDC
Three fabrics:
• 240G M2 module has maximum bandwidth
Five fabrics:
• 480G F2E/F3 module has maximum bandwidth
Fabric 2 Modules
1 Fabric 2
ASIC
2 Fabric 2
ASIC
3 Fabric 2
ASIC
4 Fabric 2
ASIC
5 Fabric 2
ASIC
per slot bandwidth
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 40
What About Nexus 7004?
• Nexus 7004 has no fabric modules
• I/O modules have local fabric with 10 available fabric channels
– I/O modules connect “back-to-back” via 8 fabric channels
– Two fabric channels “borrowed” to connect supervisor engines
Sup Slot 2 Sup Slot 1
M2/F2E/F3
Module 4
M2/F2E/F3
Module 3
Fabric
ASIC
Fabric 2
ASIC
Fabric 2
ASIC
Fabric
ASIC
2 * 55G
fabric channels
8 * 55G local fabric channels
interconnect I/O modules (440G)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 41
220Gbps 440Gbps 660Gbps 880Gbps 1100Gbps 1320Gbps Local Fab2
#1 (480G)
Local Fab2
#1 (960G)
Local Fab2
#1 (1.2T)
Fab2
#2
Fab2
#2
Fab2
#2
I/O Module Capacity – Nexus 7700
One fabric:
• Any port can pass traffic to any other port in VDC
Three fabrics:
• 480G F2E/F3 10G module has maximum bandwidth
Five fabrics:
• 960G F3 40G module has maximum bandwidth
Six fabrics:
• 1.2T F3 100G module has maximum bandwidth
per slot bandwidth
Fabric 2 Modules
1 Fabric 2
ASICs
2 Fabric 2
ASICs
3 Fabric 2
ASICs
4 Fabric 2
ASICs
5 Fabric 2
ASICs
6 Fabric 2
ASICs
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 42
Fabric, VOQ, and Arbitration
• Crossbar fabric – Provides dedicated, high-bandwidth interconnects between ingress and egress I/O modules
• Virtual Output Queues (VOQs) – Provide buffering and queuing for ingress-buffered switch architecture
• Central arbitration – Controls scheduling of traffic into fabric based on fairness, priority, and bandwidth availability at egress ports
• Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for packet transport inside switch
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 43
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 44
Buffering, Queuing, and Scheduling
• Buffering – storing packets in memory
– Needed to absorb bursts, manage congestion
• Queuing – buffering packets according to traffic class
– Provides dedicated buffer for packets of different priority
• Scheduling – controlling the order of transmission of buffered packets
– Ensures preferential treatment for packets of higher priority and fair treatment for packets of equal priority
• Nexus 7000 / Nexus 7700 use queuing policies and network-QoS policies to define buffering, queuing, and scheduling behavior
• Default queuing and network-QoS policies always in effect in absence of any user configuration
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 45
I/O Module Buffering Models
• Buffering model varies by I/O module family
– M-series modules: hybrid model combining ingress VOQ-buffered architecture with egress port-buffered architecture
– F-series modules: pure ingress VOQ-buffered architecture
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 46
e2/1 e2/12 …
Port ASIC 0
VOQ 0 VOQ 0
Supervisor Engine
Central Arbiter
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Fabric ASIC
Replication Engine 0
Port ASIC 0
Replication Engine 0
e2/1…
SP
q1
q2
q3
RE 1
VOQ 1
RE 1
VOQ 1
e1/1 e1/12
DWRR DWRR DWRR
SP
q1
q2
q3 DWRR SP DWRR SP
Fabric ASIC
k DWRR
SP
k DWRR
SP
k DWRR
SP
…
RE 2
VOQ 2
RE 3
VOQ 3
Port ASIC 1
Local
Ports
1/13 - 1/24
e1/13-24
RE 2
VOQ 2
RE 3
VOQ 3
Mo
du
le 2
Mo
du
le 1
e2/13-24
Port ASIC 1
Local
Ports
2/13 - 2/24
e1/1 - e1/11 Odd
Local
Ports
1/2 - 1/12
Even
Local
Ports
1/13 - 1/23
Odd
Local
Ports
1/14 - 1/24
Even
Local
VQIs
2/2 - 2/12
Even
Local
VQIs
2/13 - 2/23
Odd
Local
VQIs
2/14 - 2/24
Even
e2/1 – e2/11 Odd
M2 – Hybrid Ingress/Egress Buffered 10G M2 module used as example
8 ingress
queues
per port
Ingress port buffer – Manages congestion of
ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Buffering / queuing / scheduling
INGRESS QUEUING POLICIES
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 47
e2/1 e2/12 …
Port ASIC 0
VOQ 0 VOQ 0
Supervisor Engine
Central Arbiter
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Fabric ASIC
Replication Engine 0
Port ASIC 0
Replication Engine 0
e2/1…
SP
q1
q2
q3
RE 1
VOQ 1
RE 1
VOQ 1
e1/1 e1/12
DWRR DWRR DWRR
SP
q1
q2
q3 DWRR SP DWRR SP
Fabric ASIC
k DWRR
SP
k DWRR
SP
k DWRR
SP
…
RE 2
VOQ 2
RE 3
VOQ 3
Port ASIC 1
Local
Ports
1/13 - 1/24
e1/13-24
RE 2
VOQ 2
RE 3
VOQ 3
Mo
du
le 2
Mo
du
le 1
e2/13-24
Port ASIC 1
Local
Ports
2/13 - 2/24
e1/1 - e1/11 Odd
Local
Ports
1/2 - 1/12
Even
Local
Ports
1/13 - 1/23
Odd
Local
Ports
1/14 - 1/24
Even
Local
VQIs
2/2 - 2/12
Even
Local
VQIs
2/13 - 2/23
Odd
Local
VQIs
2/14 - 2/24
Even
e2/1 – e2/11 Odd
M2 – Hybrid Ingress/Egress Buffered 10G M2 module used as example
Ingress VOQ buffer – Buffers traffic
and manages congestion toward
egress destinations (VQIs)
Buffering / queuing
FABRIC-QOS POLICY
Egress VOQ buffer – Schedules
traffic toward egress destinations
(VQIs) and receives frames from fabric
Scheduling
FABRIC-QOS POLICY
VOQ buffer
carved by
source and
priority
4 priority
levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 48
e2/1 e2/12 …
Port ASIC 0
VOQ 0 VOQ 0
Supervisor Engine
Central Arbiter
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Fabric ASIC
Replication Engine 0
Port ASIC 0
Replication Engine 0
e2/1…
SP
q1
q2
q3
RE 1
VOQ 1
RE 1
VOQ 1
e1/1 e1/12
DWRR DWRR DWRR
SP
q1
q2
q3 DWRR SP DWRR SP
Fabric ASIC
k DWRR
SP
k DWRR
SP
k DWRR
SP
…
RE 2
VOQ 2
RE 3
VOQ 3
Port ASIC 1
Local
Ports
1/13 - 1/24
e1/13-24
RE 2
VOQ 2
RE 3
VOQ 3
Mo
du
le 2
Mo
du
le 1
e2/13-24
Port ASIC 1
Local
Ports
2/13 - 2/24
e1/1 - e1/11 Odd
Local
Ports
1/2 - 1/12
Even
Local
Ports
1/13 - 1/23
Odd
Local
Ports
1/14 - 1/24
Even
Local
VQIs
2/2 - 2/12
Even
Local
VQIs
2/13 - 2/23
Odd
Local
VQIs
2/14 - 2/24
Even
e2/1 – e2/11 Odd
M2 – Hybrid Ingress/Egress Buffered 10G M2 module used as example
Egress port buffer – Manages congestion at
egress physical interface
Buffering / queuing / scheduling
EGRESS QUEUING POLICIES
8 egress
queues
per port
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 49
Fabric Module 3
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric ASIC Fabric ASIC
e2/1 e2/8 …
SOC 0 SOC 2
Module 2
Supervisor Engine
Central Arbiter
Module 1
Fabric Module 1
Fabric ASIC
SOC 3
Local
Ports
1/25 - 1/32
SOC 4
Local
Ports
1/33 - 1/40
SOC 5
Local
Ports
1/41 - 1/48
SOC 0
Local
Ports
1/1 - 1/8
SOC 1
Local
Ports
1/9 - 1/16
SOC 1 SOC 1
SOC 1 SOC 1
SOC 1
Local
VQIs
2/9 – 2/16
e1/17 e1/24 e1/1-8 e1/9-16 e1/25-32 e1/33-40 e1/41-48
…
Ingress Buffer
Virtual Queuing
e2/1…
e1/17
SP
q1
q2
q3
e1/24
SP
q1
q2
q3
… Egress Buffer
DWRR SP
… e2/1 e2/8
e2/9-48
DWRR SP
F2E/F3 – Ingress Buffered
Nexus 7700 10G F3 module used as example
Ingress VOQ buffer – Buffers traffic
and manages congestion toward
egress destinations (VQIs)
Buffering / queuing
INGRESS QUEUING POLICY
Egress VOQ buffer – Schedules
traffic toward egress destinations
(VQIs) and receives frames from fabric
Scheduling
EGRESS QUEUING POLICY
4 ingress queues
per port
8 priority
levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 50
FAQ: What Is a VQI?
• VQI = Virtual Queuing Index
• “A Destination Across the Fabric”
• For M2 / F2E / F3 10G modules, VQI == 10G interface
• For F3 40/100G modules, VQI == 40/100G interface
• For M2 40/100G ports, uses multiple 10G VQIs
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 51
40G Port
Ingress Modules
10G 10G 40G 40G 100G
Spines Spines
Spines Spines Fabrics
M2 Module 40G and 100G Flow Limits
• Each Virtual Queuing Index (VQI) sustains 10G traffic flow
• All packets in given 5-tuple flow hash to single VQI
• Single-flow limit is 10G
• Packets split into 66-bit “code words”
• Four code words transmitted in parallel, one on each physical Tx fiber
• No per-flow limit imposed – splitting occurs at physical layer
Egress Interfaces
Destination
VQIs
1 VQI 1 VQI 4 VQIs 4 VQIs 10 VQIs
Internal to Nexus 7000 System
n … 4 3 2 1
64 bits
1 packet
On the Wire (40G)
Tx 1
Tx 2
Tx 3
Tx 4
66 bits
1 5
2
3
4
6
…
64
/66
B E
nco
din
g
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 52
Ingress Modules
10G 10G 40G 40G 100G
Spines Spines
Spines Spines Fabrics
F3 Module 40G and 100G Flow Limits
• Virtual Queuing Index (VQI) sustains 10G, 40G, or 100G traffic flow based on destination interface type
• No single-flow limit – full 40G/100G flow support
Egress Interfaces
Destination
VQIs
1 VQI 1 VQI 1 VQI 1 VQI 1 VQI
Internal to Nexus 7000 / 7700 System
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 53
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 54
Hardware Layer 2 Forwarding Process
Layer 2 forwarding – traffic steering based on destination MAC address
• MAC table lookup drives Layer 2 forwarding
• Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs
• Source MAC lookup drives new learns and refreshes aging timers
• Destination MAC lookup dictates outgoing switchport
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 55
Module 1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e1/1
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
Module 2
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e2/2
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
M2 L2 Packet Flow
Receive
packet from
wire
LinkSec decryption
Ingress port QoS
Submit packet
headers for
lookup
ACL/QoS/
NetFlow
lookups
VOQ arbitration
and queuing
Round-robin
transmit to fabric
Receive from
fabric
Return buffer
credit
Return
credit
to pool
Transmit
packet on
wire
Return result –
destination +
hash result
Credit grant for
fabric access
Egress
port QoS LinkSec
encryption
Static or hash-
based RE uplink
selection
Hash-based uplink
and VQI selection
Round-robin
transmit to VQI
Static
downlink
selection
L2 SMAC/ DMAC
lookups
Port-channel hash
result
HDR = Packet Headers DATA = Packet Data = Internal Signaling CTRL
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 56
SoC
VOQ
SoC
DE
F2E / F3 L2 Packet Flow
Module 2
Fabric ASIC
e2/2
Module 1
Fabric ASIC
e1/1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
VOQ arbitration
Credit grant for
fabric access
Receive from fabric
Return
credit
to pool
Transmit
packet on
wire
Fabric Module 4
Fabric ASIC
Fabric Module 5
Fabric ASIC
Transmit
to fabric
VOQ
Receive
packet
from wire
Ingress
port QoS
(VOQ)
Ingress L2 SMAC/ DMAC
lookups, ACL/QoS lookups,
NetFlow sampling Return result –
destination
Submit packet headers for lookup
Egress port QoS
(Scheduling)
Return buffer credit
HDR = Packet Headers DATA = Packet Data = Internal Signaling CTRL
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 57
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 58
Layer 3 Forwarding
• Nexus 7000 decouples control plane and data plane
• Forwarding tables built on control plane using routing protocols or static configuration
–OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing
• Tables downloaded to forwarding engine hardware for data plane forwarding
–FIB TCAM contains IP prefixes
–Adjacency table contains next-hop information
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 59
Hardware Layer 3 Forwarding Process
• FIB TCAM lookup based on longest-match destination prefix comparison
• FIB “hit” returns adjacency, adjacency contains rewrite information (next-hop)
• Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 60
10.1.1.2
10.1.1.3
10.10.0.10
10.10.0.100
10.10.0.33
10.1.1.4
10.1.2.xx
10.1.3.xx
10.1.1.xx
10.100.1.xx
10.10.0.xx
10.100.1.xx
10.10.100.xx
IP FIB TCAM Lookup
FIB TCAM
Generate
Lookup Key
10.1.1.10
Generate TCAM lookup key
(destination IP address)
Forwarding Engine
FIB DRAM
Load-Sharing Hash
Adjacency Table
Next-hop 4 (IF, MAC)
Next-hop 6 (IF, MAC)
Next-hop 7 (IF, MAC)
Next-hop 5 (IF, MAC)
Next-hop 3 (IF, MAC)
Next-hop 1 (IF, MAC)
Next-hop 2 (IF, MAC)
10.1.1.xx
Ingress
unicast IP
packet header
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Hit in FIB
returns result
in FIB DRAM
Adjacency
index identifies
ADJ block to
use
Modulo function
selects exact
next hop entry
to use
Offset
Compare
lookup key
Return lookup
result
# next-
hops
Flow
Data
Result HIT!
Adj Index
mod
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 61
Module 1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e1/1
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
Module 2
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e2/2
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
M2 L3 Packet Flow
Receive
packet from
wire
LinkSec decryption
Ingress port QoS
Submit packet
headers for
lookup
L3 FIB/ADJ lookup
Ingress and egress
ACL/QoS/NetFlow
lookups
VOQ arbitration
and queuing
Round-robin
transmit to fabric
Receive from
fabric
Return buffer
credit
Return
credit
to pool
Transmit
packet on
wire
Return result –
destination +
hash result
Credit grant for
fabric access
Egress
port QoS LinkSec
encryption
Static or Hash-based
uplink selection
Hash-based uplink
(and VQI) selection
Round-robin
transmit to VOQ
Static RE
downlink
selection
L2 ingress and egress
SMAC/ DMAC lookups
Port-channel hash result
HDR = Packet Headers DATA = Packet Data = Internal Signaling CTRL
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 62
SoC
VOQ
SoC
DE
Module 2
Fabric ASIC
e2/2
Module 1
Fabric ASIC
e1/1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric Module 4
Fabric ASIC
Fabric Module 5
Fabric ASIC
VOQ
F2E / F3 L3 Packet Flow HDR = Packet Headers DATA = Packet Data = Internal Signaling CTRL
VOQ arbitration
Credit grant for
fabric access
Return
credit
to pool
Transmit
packet on
wire
Transmit
to fabric
Receive
packet
from wire
Ingress
port QoS
(VOQ)
Return result –
destination
Submit packet headers for lookup
L2 ingress and egress SMAC/
DMAC lookups
L3 FIB/ADJ lookup
Ingress and egress ACL/QoS
lookups, NetFlow sampling
Receive from fabric
Egress port QoS
(Scheduling)
Return buffer credit
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 63
Layer 3 Forwarding – Module Interoperability Models
Two interoperability models for L3 forwarding:
• “Proxy Forwarding”
• “Ingress Forwarding” with Lowest Common Denominator
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 64
Proxy Forwarding Model – Conceptual
• From F2E perspective, Router MAC reachable through giant port-channel
• All packets destined to Router MAC forwarded through fabric toward one “member port” in that channel
All F2E modules
All M2 modules
Up to 128 “links” 10.1.10.100 vlan 10
10.1.20.100 vlan 20
interface vlan 10
ip address 10.1.10.1/24
!
interface vlan 20
ip address 10.1.20.1/24
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 65
Proxy Forwarding Model – Actual
10.1.10.100 vlan 10
e1/1 Fabric
F2E
SoC
FE
e2/1 Fabric
F2E
SoC
FE
10.1.20.100 vlan 20
Replication
Engine
e3/1
e3/2
M2
Replication
Engine
Replication
Engine
Replication
Engine
VOQs
VOQs
FE
FE
Fabric
e3/7 e3/8
e3/13
e3/14
e3/19 e3/20
Replication
Engine
e4/1
e4/2
M2
Replication
Engine
Replication
Engine
Replication
Engine
VOQs
VOQs
FE
FE
Fabric
e4/7
e4/8
e4/13
e4/14
e4/19
e4/20
Fabric
Fabric Modules
Fabric
…
VLAN DMAC Dest Port
10 router_mac → internal_channel (e3/1-24,e4/1-24)
EtherChannel Hash Function
hash_input (from packet) → select_member_port
Ingress MAC:
VLAN DMAC Dest Port
10 router_mac → L3_lookup
Routing:
DIP Next Hop
10.1.20.100 → server_2_mac (v20)
Egress MAC:
VLAN DMAC Dest Port
20 server_2_mac → e2/1
1
2
3
4
6
5 7
8
9
10
Programming of all M2 forwarding engines
Programming of all F2E forwarding engines
interface vlan 10
ip address 10.1.10.1/24
!
interface vlan 20
ip address 10.1.20.1/24
Can be up to 128 M2 VQIs
Mod 1
Mod 2
Mod 4
Mod 3
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 66
Proxy Layer 3 Forwarding – Process
1. Host connected to F2E module (e1/1) in VLAN 10 wants to send traffic to host connected to F2E module (e2/1) in VLAN 20 – host sends traffic to DMAC of VLAN 10 SVI (after ARPing for gateway)
2. SoC ASIC on F module does DMAC lookup for Router MAC, output “port” for such traffic is internal L3 port-channel comprised of all the M1/M2 front-panel ports (i.e., router can be reached out any of those ports)
3. Packet hashes to one of the available links (in this case it hashes to e3/6)
4. Packet is sent across the fabric toward e3/6; Replication engine owning e3/6 receives packet from fabric, but packet is treated as if it was received from the wire
5. Replication engine sends packet headers to M forwarding engine
6. Ingress MAC lookup hits Router MAC entry, triggering full L3 lookup
7. Dest IP lookup returns next hop of 10.1.20.100 (Server 2 is directly connected) with DMAC “server_2_mac”
8. Egress MAC lookup hits server_2_mac entry, returning egress port where Server 2 is connected (e2/1)
9. Forwarding engine returns result to replication engine, indicating output VLAN, rewrite MAC addresses, and output port (e2/1)
10. Packet is sent back across the fabric to module 2, which forwards it out e2/1
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 67
Ingress Forwarding with Lowest Common Denominator Model • F3 module interoperability always Ingress Forwarding – NO proxy forwarding
with F3
– The ingress module makes all the forwarding decisions
• Supported feature set based on Lowest Common Denominator
– Feature available if all modules support the feature
VDC Type Layer 2 Layer 3 vPC Fabric
Path VXLAN FEX MPLS OTV LISP FCoE Table Sizes
F3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ F3 size
M2 + F3 ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✗ ✗ F3 size
F2/F2E + F3 ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✓ F2E size
Not all features
supported by
software today
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 68
Interoperability Forwarding Model Matrix
Module Combination Interoperability Model
M1 + M2 Lowest Common Denominator
M + F1 Proxy Forwarding
M + F2E Proxy Forwarding
F2 + F2E Lowest Common Denominator
F2 + F2E + F3 Lowest Common Denominator
M2 + F3 Lowest Common Denominator
M2 + F2/F2E + F3 NOT SUPPORTED
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 69
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 70
What Is Classification?
• Matching packets
– Layer 2, Layer 3, and/or Layer 4 information
• Used to decide whether to apply a particular policy to a packet
– Enforce security, QoS, or other policies
• Some examples:
– Match TCP/UDP source/destination port numbers to enforce security policy
– Match destination IP addresses to apply policy-based routing (PBR)
– Match 5-tuple to apply marking policy
– Match protocol-type to apply Control Plane Policing (CoPP)
– etc.
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 71
CL TCAM Lookup – ACL ip access-list example
permit ip any host 10.1.2.100
deny ip any host 10.1.68.44
deny ip any host 10.33.2.25
permit tcp any any eq 22
deny tcp any any eq 23
deny udp any any eq 514
permit tcp any any eq 80
permit udp any any eq 161
xxxxxxx | 10.1.2.100 | xx | xxx | xxx
xxxxxxx | 10.1.68.44 | xx | xxx | xxx
xxxxxxx | 10.33.2.25 | xx | xxx | xxx
xxxxxxx | xxxxxxx | tcp | xxx | 22
xxxxxxx | xxxxxxx | tcp | xxx | 23
xxxxxxx | xxxxxxx | tcp | xxx | 80
xxxxxxx | xxxxxxx | udp | xxx | 161
xxxxxxx | xxxxxxx | udp | xxx | 514
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
CL TCAM
Generate
Lookup Key
Generate TCAM
lookup key
CL SRAM
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
xxxxxxx | 10.2.2.2 | xx | xxx | xxx
xxxxxxx | xxxxxxx | tcp | xxx | 80
SIP | DIP | Pr | SP | DP
Compare lookup
key to CL TCAM
entries
Comparisons (X = “Mask”)
Hit in CL TCAM
returns result in
CL SRAM
Security ACL
Forwarding Engine
Result
Return
lookup
result
Result affects
final packet
handling
Permit
Permit
Permit
Permit
Deny
Deny
Deny
Deny
HIT!
Results
SIP | DIP | Pr | SP | DP
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 72
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
Result affects
final packet
handling
Generate
Lookup Key
Forwarding Engine
xxxxxxx | 10.3.3.xx | xx | xxx | xxx
xxxxxxx | 10.4.12.xx | xx | xxx | xxx
10.1.1.xx | xxxxxxx | udp | xxx | xxx
10.1.1.xx | xxxxxxx | tcp | xxx | xxx
xxxxxxx | 10.5.5.xx| tcp | xxx | 23
CL TCAM Lookup – QoS ip access-list police
permit ip any 10.3.3.0/24
permit ip any 10.4.12.0/24
ip access-list remark-dscp-32
permit udp 10.1.1.0/24 any
ip access-list remark-dscp-40
permit tcp 10.1.1.0/24 any
ip access-list remark-prec-3
permit tcp any 10.5.5.0/24 eq 23
CL TCAM
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
xxxxxxx | 10.2.2.xx | xx | xxx | xxx
10.1.1.xx | xxxxxxx | tcp | xxx| xxx
HIT!
CL SRAM
QoS Classification ACLs
Generate
TCAM lookup
key
SIP | DIP | Pr | SP | DP
Compare
lookup
key
Hit in CL TCAM
returns result in
CL SRAM
Result
Return
lookup
result
Policer ID 1
Policer ID 1
Remark DSCP 32
Remark DSCP 40
Remark IP Prec 3
SIP | DIP | Pr | SP | DP
Comparisons (X = “Mask”)
Results
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 73
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 74
NetFlow
• NetFlow collects flow data for packets traversing the switch
• Each module maintains independent NetFlow table
M2 F2E / F3
Per-interface NetFlow Yes Yes
NetFlow direction Ingress/Egress Ingress only
Full NetFlow Yes No
Sampled NetFlow Yes Yes
FSA Assist for Sampled NetFlow No F3 only (future)
Bridged NetFlow Yes Yes
Hardware Cache Yes No
Software Cache No Yes
Hardware Cache Size 512K entries per
forwarding engine N/A
NDE (v5/v9) Yes Yes
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 75
Full vs. Sampled NetFlow
• NetFlow collects full or sampled flow data
• Full NetFlow: Accounts for every packet of every flow on interface
– Available on M-Series modules only
– Flow data collection up to capacity of hardware NetFlow table
• Sampled NetFlow: Accounts for M in N packets on interface
– Available on both M2 (ingress/egress) and F2E/F3 (ingress only)
– M2: Flow data collection up to capacity of hardware NetFlow table
– F2E/F3: Flow data collection for up to ~1000/3000pps per module
– F3 (future): Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 76
Sampled NetFlow Details
• Random packet-based sampling
• M:N sampling: Out of N consecutive packets, select M consecutive packets and account only for those flows
• On M2, sampled packets create hardware NetFlow table entry
• On F2E/F3, sampled packets sent to LC CPU via module inband
– Rate limited to ~1000pps per module
• Software multiplies configured sampler rate by 100 on F2E/F3 modules
– Example: when using 1 out-of 100 sampler on F2/F2E interface, sampled rate becomes 1:10000
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 77
NetFlow on M2 Modules
Fabric
ASIC
VOQs
Mgmt Enet
Supervisor
Engine
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Hardware
Flow Creation
Hardware
Flow Creation
Hardware
Flow Creation
Aged Flow Info
Aged Flow Info
Aged Flow Info
Generate NetFlow v5
or v9 export packets
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via Supervisor
Inband
via mgmt0
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 78
Sampled NetFlow on F2E/F3 Modules
F3 Module
FSA
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
F3 Module
Fabric
ASIC
VOQs
Mgmt Enet
Supervisor
Engine
FSA
CPU
SoC
Decision
Engine
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via Supervisor
Inband
via mgmt0
DRAM
NetFlow
Cache
Populate cache based
on received samples
Age flows and
generate NetFlow v5
or v9 export packets
F2E Module
LC
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
Data Flow
Data Flow
Data Flow
via Module
Inband
via Module
Inband
via Module
Inband
Sampled
Packets
Sampled
Packets
Sampled
Packets
Aged
Flows
Aged
Flows
Aged
Flows
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 79
Agenda
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Forwarding Engine Architecture
• Fabric Architecture
• I/O Module Queuing
• Layer 2 Forwarding
• Layer 3 Forwarding
• Classification
• NetFlow
• Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 80
Nexus 7000 / Nexus 7700 Architecture Summary
I/O Modules
Supervisor Engines
Fabrics
Chassis
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 81
Conclusion
• You should now have a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, I/O module design, packet flows, and key forwarding engine functions…
• Any questions?
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 82
Reference: Acronym Decoder
• ACL–Access Control List
• ADJ–Adjacency
• ASIC–Application Specific Integrated Circuit
• CFP–C Formfactor Pluggable
• CoPP–Control Plane Policing
• COS–Class of Service
• DSCP–Differentiated Services Code Point
• DWRR–Deficit Weighted Round Robin
• ECMP–Equal Cost Multi Path
• EEE–Energy Efficient Ethernet
• EOBC–Ethernet Out-of-Band Channel
• FCoE–Fiber Channel over Ethernet
• FE–Forwarding Engine
• FEX–Fabric Extender (Nexus 2000 family)
• FIB–Forwarding Information Base
• FRU–Field Replaceable Unit
• GRE–Generic Route Encapsulation
• HSRP–Hot Standby Router Protocol
• IGMP–Internet Group Management Protocol
• MPLS–Multiprotocol Label Switching
• NDE–NetFlow Data Export
• OTV–Overlay Transport Virtualization
• PACL–Port ACL
• PBR–Policy-Based Routing
• PIM–Protocol Independent Multicast
• QoS–Quality of Service
• QSFP+–40G Quad Small-Formfactor Pluggable
• RACL–Router ACL
• RE–Replication Engine
• RPF–Reverse Path Forwarding
• RU–Rack Unit
• SFP+–10G Small-Formfactor Pluggable
• SoC–System-on-chip/switch-on-chip
• TCAM–Ternary CAM
• uRPF–Unicast RPF
• VACL–VLAN ACL
• VDC–Virtual Device Context
• VOQ–Virtual Output Queuing
• VQI–Virtual Queuing Index
• XL–Refers to forwarding engine with larger FIB and ACL TCAMs
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 83
Complete Your Online Session Evaluation
• Give us your feedback and you could win fabulous prizes. Winners announced daily.
• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
83
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 84
Continue Your Education
• Demos in the Cisco Campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
84