Cisco Nexus 7000 H/W Architecture

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKARC-3470 1

BRKARC-3470

Cisco Nexus 7000 Hardware Architecture


Session Goal

To provide you with a thorough understanding of the Cisco Nexus™ 7000 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions

This session will not examine NX-OS software architecture or other Nexus platform architectures

Related sessions:BRKARC-3471: Cisco NX-OS Software Architecture

BRKDCT-2951: Deploying Nexus 7000 in Data Center Networks

BRKDCT-2081: Cisco FabricPath Technology and Design

BRKDCT-2048: Deploying Virtual Port Channel in NX-OS

BRKDCT-2121: VDC Design and Implementation Considerations with Nexus 7000

BRKARC-3472: NX-OS Routing & Layer 3 Switching

BRKCRS-3144: Troubleshooting Cisco Nexus 7000 Series Switches

TECDCT-4125: Cisco FabricPath (4h Techtorial)

LTRDCT-4047: Deploying Nexus 7000/NX-OS Hands-on Lab (Lab)

LTRCRT-5205: Configuring Nexus 7000 Virtualization (Lab)

2


What Is Nexus 7000?

Data-center class Ethernet switch designed to deliver high-availability, system scale, usability, investment protection

I/O Modules

Forwarding Engines

Supervisor Engines

Fabrics

Chassis


Agenda

Chassis Architecture

Supervisor Engine and I/O Module Architecture

Forwarding Engine Architecture

Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding

IP Multicast Forwarding

Classification Policies

NetFlow

Summary


Nexus 7010 Chassis

Optional

front door

Front Rear

System status

LEDs

Integrated cable

management

with cover

Supervisor

slots (5-6)

I/O module slots

(1-4, 7-10)

Air intake with

optional filter

Air exhaust

Crossbar fabric

modules

Fan trays

Power supplies

21RU

Front-to-

back airflow

Two chassis

per 7’ rack

N7K-C7010


Nexus 7018 Chassis

Front Rear

System status

LEDs

Integrated cable

management

Supervisor

slots (9-10)

Power supply

air intake

Crossbar

fabric

modules

Power supplies

25RU

Side-to-side

airflow

Fan trays

I/O module slots

(1-8, 11-18)

Optional front

door

Supported in NX-OS release 4.1(2) and later

N7K-C7018


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


Supervisor Engine

Performs control plane and management functions

Dual-core 1.66GHz x86 processor with 8GB DRAM

2MB NVRAM, 2GB internal bootdisk, compact flash slots, USB

Console, aux, and out-of-band management interfaces

Interfaces with I/O modules via 1G switched EOBC

Houses dedicated central arbiter ASIC that controls VOQ admission/fabric access via dedicated arbitration path to I/O modules

N7K-SUP1

ID LED

Console Port

AUX Port

Management

Ethernet

USB Ports CMP Ethernet

Reset ButtonStatus

LEDs

Compact Flash

Slots


Nexus 7000 I/O Module Families – M1 and F1

M family – L2/L3/L4 with large forwarding tables and rich feature set

F family – Low-cost, high performance, low latency, low power and streamlined feature set

N7K-M108X2-12L

N7K-M132XP-12/N7K-M132XP-12L

N7K-M148GT-11/N7K-M148GT-11L

N7K-M148GS-11/N7K-M148GS-11L

N7K-F132XP-15


8-Port 10GE M1 I/O ModuleN7K-M108X2-12L

8-port 10G with X2 transceivers

80G full-duplex fabric connectivity

Two integrated forwarding engines (120Mpps)

Support for “XL” forwarding tables (licensed feature)

Distributed L3 multicast replication

802.1AE LinkSec

N7K-M108X2-12L

Supported in NX-OS release 5.0(2a) and later


8-Port 10G XL M1 I/O Module ArchitectureN7K-M108X2-12L

1

Forwarding

Engine

VOQs VOQs

Fabric ASIC

To Fabric Modules

10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC

Replication

Engine

Replication

Engine

Replication

Engine

Replication

Engine

Front Panel Ports

Forwarding

Engine

2 3 4 5 6 7 8

LC

CPU

EOBC To Central Arbiter

Linksec Linksec Linksec Linksec Linksec Linksec Linksec Linksec


32-Port 10GE M1 I/O ModulesN7K-M132XP-12, N7K-M132XP-12L

32-port 10G with SFP+ transceivers

80G full-duplex fabric connectivity

Integrated 60Mpps forwarding engine

XL forwarding engine on “L” version

Oversubscription option for higher density (up to 4:1)

Supports Nexus 2000 (FEX) connections


802.1AE LinkSec

N7K-M132XP-12/

N7K-M132XP-12L

Cisco Public 13

N7K-M132XP-12 – Supported in all releases

N7K-M132XP-12L – Supported in NX-OS release 5.1(1) and later


Shared vs. Dedicated Mode

9 11 13 15

9 11 13 15

Dedicated mode

First interface in port groupgets 10G bandwidth

Other three interfaces in portgroup disabled

Shared mode

Four interfaces in port group share 10G bandwidth

10G

To Fabric

10G

To Fabric

“Port group”– group of contiguous even or odd ports that share 10G of bandwidth (e.g., ports 1,3,5,7)

rate-mode shared

(default)

rate-mode dedicated


1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

32-Port 10G M1 I/O Module ArchitectureN7K-M132XP-12, N7K-M132XP-12L

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

Forwarding EngineVOQs VOQs

Fabric ASIC

To Fabric Modules

10G MAC 10G MAC 10G MAC 10G MAC

4:1 Mux +

Linksec

4:1 Mux +

Linksec

4:1 Mux +

Linksec

4:1 Mux +

Linksec

4:1 Mux +

Linksec

4:1 Mux +

Linksec

4:1 Mux +

Linksec

4:1 Mux +

Linksec

10G MAC 10G MAC 10G MAC 10G MAC

Replication

Engine

Replication

Engine

Replication

Engine

Replication

Engine

Front Panel Ports

LC

CPU



N7K-M148GT-11All releases

N7K-M148GT-11LRelease 5.1(1) and later

48-Port 1G M1 I/O ModulesN7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L

Four 1G I/O module options:

48 10/100/1000 RJ-45 ports (N7K-M148GT-11)

48 1G SFP ports (N7K-M148GS-11)

48 1G SFP ports with XL forwarding engine (N7K-M148GS-11L)

48 10/100/1000 RJ-45 ports with XL forwarding engine (N7K-M148GT-11L)

Integrated 60Mpps forwarding engine

46G full duplex fabric connectivity

Line rate on 48-ports with some local switching


802.1AE LinkSec

N7K-M148GS-11Release 4.1(2) and later

N7K-M148GS-11LRelease 5.0(2a) and later


48-Port 1G M1 I/O Modules ArchitectureN7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L

12 x 1G MAC

Fabric ASIC

VOQs

12 x 1G MAC 12 x 1G MAC12 x 1G MAC

1-12

Front Panel Ports

13-24 25-36 37-48

Forwarding Engine

To Fabric Modules

Replication

Engine

Replication

Engine

LC

CPU


Linksec Linksec Linksec Linksec Linksec Linksec


32-Port 1G/10GE F1 I/O ModuleN7K-F132XP-15

32-port 1G/10G with SFP/SFP+ transceivers

230G full-duplex fabric connectivity (320G local switching)

System-on-chip (SoC)† forwarding engine design

16 independent SoC ASICs

Layer 2 forwarding with L3/L4 services (ACL/QoS)

Multi-protocol – Classic Ethernet, FabricPath, DCB, FCoE

N7K-F132XP-15

Supported in NX-OS release 5.1(1) and later

† sometimes called “switch-on-chip”


32-Port 1G/10G F1 I/O Module ArchitectureN7K-F132XP-15

1

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

Front Panel Ports

Fabric ASIC

To Fabric Modules

Fabric ASIC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

LC

CPU


Arbitration

Aggregator …


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


M1 Forwarding Engine Hardware

Hardware forwarding engine(s) integrated on every I/O module

60Mpps per forwarding engine Layer 2 bridging with hardware MAC learning

60Mpps per forwarding engine IPv4 and 30Mpps IPv6 unicast

IPv4 and IPv6 multicast support (SM, SSM, bidir)

RACL/VACL/PACLs

QoS remarking and policing policies

Policy-based routing (PBR)

Unicast RPF check and IP source guard

Ingress and egress NetFlow (full and sampled)

Hardware Table M1 Modules M1-XL Modules without License

M1-XL Modules with License

FIB TCAM 128K 128K 900K

Classification TCAM (ACL/QoS) 64K 64K 128K

MAC Address Table 128K 128K 128K

NetFlow Table 512K 512K 512K


Layer 3

Engine

M1 Forwarding Engine ArchitectureFE Daughter

Card

Layer 2

Engine

Packet Headers from

I/O Module Replication Engine

Final lookup result to

I/O Module Replication Engine

Ingress MAC

table lookups

IGMP snooping

lookups

IGMP snooping

redirection

Egress MAC

lookups

IGMP snooping

lookups

FIB TCAM and

adjacency table

lookups for Layer

3 forwarding

ECMP hashing

Multicast RPF

checkUnicast RPF

check

Ingress ACL

and QoS

classification

Ingress NetFlow

collection

Egress NetFlow

collection

Egress Pipeline

Ingress Pipeline

Egress ACL and QoS classification

Ingress policing

Egress policing


F1 Forwarding Engine Hardware

SoC forwarding engine services two front-panel 10G ports

16 SoCs per I/O module

480Mpps Layer 2 bridging with hardware MAC learning per I/O module

30Mpps per forwarding engine

VACL/PACLs

QoS remarking policies

FabricPath forwarding

Priority Flow-Control (PFC) and Enhanced Transmission Selection (ETS)

Hardware Table Per SoC Per F1 Module

MAC Address Table 16K 256K

Classification TCAM (ACL/QoS) 1K in/1K out 16K in/16K out


Port A

1G/10G

Port B

1G/10G

Forwarding Engine

(Ingress)

Ingress Buffer

(VOQ)

Forwarding Engine

(Egress)

Egress Buffer

To Fabric

To/From Central

Arbiter From Fabric

2 X 10G

SoC

1G/10G MAC

MAC Table,

ACL, QoS

Pre-Forwarding

Ingress Buffer

1G/10G MAC

F1 Forwarding Engine Architecture

1G and 10G

capable interface

MAC

Two front-panel

interfaces per

ASIC

“Skid buffer” –

Accommodates pause

reaction time

Ingress forwarding

decision (MAC

lookup, FP lookup,

ingress ACL/QoS

Egress forwarding

decision (MAC lookup,

egress ACL)

Virtual output

queues

Egress fabric

receive buffer

Forwarding

tables


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


Crossbar Switch Fabric Module

Each fabric module provides 46Gbps per I/O module slot

Up to 230Gbps per slot with 5 fabric modules

Different I/O modules leverage different amount of fabric bandwidth

80G per slot with 10G M1 modules

230G per slot with 10G F1 modules

Access to fabric controlled using QoS-aware central arbitration with VOQ

N7K-C7018-FAB-1

N7K-C7010-FAB-1


1

Nexus 7000 implements 3-stage crossbar switch fabric

Stages 1 and 3 on I/O modules

Stage 2 on fabric modules

Multistage Crossbar

1st stage Egress

Module

2nd stage

Ingress

Module

3rd stage

2 x 23Gbps per I/O slot

per fabric module

Up to 230Gbps per I/O

module with 5 fabric

modules installed

20 x 23Gbps

channels per

fabric module

Crossbar

Fabric

ASIC

2 3 4 5Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Crossbar

Fabric

ASIC

Fabric Modules


M1 I/O Module Capacity

4th and 5th fabric modules provide additional redundancy and future-proofing

Require 2 fabrics for N+1 redundancy

1G modules Require 1 fabric for full

bandwidth

10G modules

Require 3 fabrics for N+1 redundancy

Require 2 fabrics for full bandwidth

46Gbps/slot

46Gbps/slot

46Gbps/slot

46Gbps/slot

46Gbps/slot

Fabric Modules

1Crossbar

Fabric

ASICs

2Crossbar

Fabric

ASICs

3Crossbar

Fabric

ASICs

4Crossbar

Fabric

ASICs

5Crossbar

Fabric

ASICs

46Gbps92Gbps138Gbps184Gbps230Gbpsper slot bandwidth


46Gbps92Gbps138Gbps184Gbps230Gbps

230G

F1 I/O Module CapacityFabric Modules

230G

2Crossbar

Fabric

ASICs

3Crossbar

Fabric

ASICs

4Crossbar

Fabric

ASICs

5Crossbar

Fabric

ASICs

per slot bandwidth

Requires 5 fabrics for maximum bandwidth

F1 SFP+ module Operates with any number

of fabrics

46Gbps/slot

46Gbps/slot

46Gbps/slot

Redundancy model is graceful bandwidth derating

1Crossbar

Fabric

ASICs46Gbps/slot

46Gbps/slot


Access to Fabric Bandwidth

Access to fabric controlled using central arbitration

Arbiter ASIC on Supervisor Engine provides fabric arbitrationvia dedicated arbitration path

Virtual Output Queues (VOQs) at ingress to fabric represent bandwidth availability on egress modules

Four levels of priority per VOQ destination

Central arbiter controls admission to VOQ based on bandwidth availability and priority

“Buffer credits” represent bandwidth availability for each VOQ destination at each priority level

Credits requested by ingress I/O modules with traffic to send into fabric

Credits granted by central arbiter based on bandwidth availability

Credits returned to the “pool” by egress I/O modules after receiving traffic from fabric


Benefits of Central Arbitration with VOQ

Ensures priority traffic takes precedence over best-effort traffic across fabric

Four levels of priority for each VOQ destination

Ensures fair access to bandwidth for multiple ingress ports transmitting to one egress port

Central arbiter ensures all traffic sources get appropriate access to fabric bandwidth, even with traffic sources on different modules

Prevents congested egress ports from blocking ingress traffic destined to other ports

Mitigates head-of-line blocking by providing independent queues for individual destinations across the fabric

Enables lossless service for some traffic classes across the fabric

Can provide strict priority and backpressure (blocking instead of dropping) for certain traffic classes, such as FCoE traffic

34


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


I/O Module Queuing

All configuration through Modular QoS CLI (MQC)

Queuing parameters applied using class-maps/policy-maps/service-policies

Queuing and scheduling behavior based on I/O module capabilities

Default queuing on all modules enables two classes including egress priority queue

Buffering model varies by I/O module family

M1 modules: hybrid model combining egress port buffered with ingress VOQ buffered architecture

F1 modules: pure ingress VOQ buffered architecture


Hybrid Ingress/Egress Buffered Model (M1 I/O Modules)

Crossbar

Fabric

Ingress Module

Ingress Module

Ingress Module

Egress Module

Ingress

port buffer

Ingress

VOQ buffer

Egress

port buffer

Ingress port buffer – manages congestion in ingress forwarding/replication engines only

Ingress VOQ buffer – manages congestion toward egress interface over fabric

Egress FIFO buffer – just enough to catch frames “in flight” and keep pipe full

Egress port buffer – manages congestion at egress interface

Egress

FIFO buffer


Ingress Buffered Model (F1 I/O Modules)

Crossbar

Fabric

Ingress Module

Ingress Module

Ingress Module

Egress Module

Ingress

skid buffer

Ingress

VOQ buffer

Egress

FIFO buffer

Ingress “skid” buffer – just enough to absorb packets received after external flow control asserted

Ingress VOQ buffer – manages congestion “toward egress interfaces”

Egress FIFO buffer – just enough to catch frames “in flight” and keep pipe full


VOQs

Replication

Engine

10G MAC10G MAC

Port 1 Port 2

To fabric

1

Forwarding

Engine

VOQs VOQs

Fabric ASIC

10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC

ReplicationEngine

ReplicationEngine

ReplicationEngine

ReplicationEngine

Front Panel Ports

Forwarding

Engine

2 3 4 5 6 7 8

LCCPU

Linksec Linksec Linksec Linksec Linksec Linksec Linksec Linksec

M1 I/O Module Buffering

Hybrid ingress/egress buffered architecture on 8-port 10G M1 I/O module

Ingress96MB1 2 3 4 5 6 7 8

8q2t

Egress80MB

1p7q4t

1 2 3 4 5 6 7 8

1p3q1t

1 2 3 4VOQ

16MB

Per-Port

Shared

Egress

FIFO

1


Port 1 Port 2

2 X 10G

SoC

To fabric

1

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

Front Panel Ports

Fabric ASICFabric ASIC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 X 10G

SoC

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

LC

CPU

Arbitration

Aggregator…

VOQ1.25MB

F1 I/O Module Buffering

Pure ingress buffered architecture on 32-port 10G F1 I/O module

Skid

Per-Port

Egress

FIFO

1

1p3q1t

1 2 3 4

FIFO

1


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


Layer 2 Forwarding

MAC table size depends on module type:

M1 MAC table is 128K entries

F1 MAC table is 256K entries (16K entries per SoC)

Hardware MAC learning

CPU not directly involved in learning

Forwarding engine(s) on each module have copy of MAC table

New learns communicated to other forwarding engines via hardware “flood to fabric” mechanism

Software process ensures continuous MAC table sync

Spanning tree (PVRST or MST), Virtual Port Channel (VPC), or FabricPath ensures loop-free Layer 2 topology


Hardware Layer 2 Forwarding Process

In Classic Ethernet and FabricPath edge switches, MAC table lookup drives Layer 2 forwarding

Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs

Source MAC lookup drives new learns and refreshes aging timers

Destination MAC lookup dictates outgoingswitchport (CE/FabricPath local) or destination Switch ID (FabricPath remote)

In FabricPath core switches, Switch ID (routing) table lookup drives Layer 2 forwarding

Destination SID lookup dictates outgoing FabricPath interface and next hop


Layer 2

Engine

Layer 3

Engine

Forwarding

Engine

Module 1

VOQs

10G MAC

Replication

Engine

Fabric ASIC

Linksec

e1/1

Layer 2

Engine

Layer 3

Engine

Forwarding

Engine

Module 2

VOQs

10G MAC

Linksec

Replication

Engine

Fabric ASIC

e2/1

Fabric Module 1

Fabric ASIC

Fabric Module 2

Fabric ASIC

Fabric Module 3

Fabric ASIC

M1 L2 Packet Flow Supervisor Engine

Central Arbiter

HDR = Packet Headers DATA = Packet Data = Internal SignalingCTRL

Receive

packet

from wire

LinkSec decryption

Ingress port QoS

Submit packet

headers for

lookup

L2 SMAC/

DMAC lookups

ACL/QoS/

NetFlow

lookups

VOQ arbitration

and queuing

Transmit

to fabric

Receive from

fabric

Return buffer

credit

Return

credit

to pool

Egress

port QoS LinkSec

encryption

Transmit

packet on

wire

Return result

Credit grant

for fabric

access

3

1

2

4

5

6

7

8

9

10

11

12

13

14

15e2/1


F1 L2 Packet Flow

Module 2

Fabric ASIC

e2/1

SoC

e2/2

Module 1

Fabric ASIC

e1/1

SoC

e1/2

Fabric Module 1

Fabric ASIC

Fabric Module 2

Fabric ASIC

Fabric Module 3

Fabric ASIC

Supervisor Engine

Central Arbiter


Receive

packet

from wire

1

Ingress L2 & ACL/QoS

lookups

2

VOQ arbitration

and queuing

3

Credit grant

for fabric

access 4

Transmit

to fabric

5 Receive from

fabric

Return buffer

credit

Return

credit

to pool

6

7

Transmit

packet on

wire8


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


IP Forwarding

Nexus 7000 decouples control plane and data plane

Forwarding tables built on control plane using routing protocols or static configuration

OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing

Tables downloaded to forwarding engine hardware for data plane forwarding


I/O ModuleI/O Module I/O Module

Supervisor Engine

BGP OSPF ISIS RIP EIGRP

URIB/U6RIB

UFDM

Hardware

FIB TCAM ADJ Table

IP Forwarding Architecture

Routing protocol processes learn routing information from neighbors

IPv4 and IPv6 unicast RIBs calculate routing/next-hop information

Unicast Forwarding Distribution Manager (UFDM) interfaces between URIBs on supervisor and IP FIB on I/O modules

IP FIB process programs forwarding engine hardware on I/O modules

FIB TCAM contains IP prefixes

Adjacency table contains next-hop information

Hardware

IP FIB

HardwareHardware

IP FIB IP FIB

n7010# sh processes cpu | egrep ospf|PID

PID Runtime(ms) Invoked uSecs 1Sec Process

20944 93 33386880 0 0 ospf

n7010# sh processes cpu | egrep u.?rib

3573 117 44722390 0 0 u6rib

3574 150 34200830 0 0 urib

n7010# sh processes cpu | egrep ufdm

3836 1272 743933460 0 0 ufdm

module-9# sh processes cpu | egrep fib

1534 80042 330725 242 0.0 ipfib

module-9#


Hardware IP Forwarding Process

FIB TCAM lookup based on destination prefix (longest-match)

FIB “hit” returns adjacency, adjacency contains rewrite information (next-hop)

Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result


10.1.1.2

10.1.1.3

10.10.0.10

10.10.0.100

10.10.0.33

10.1.1.4

10.1.2.xx

10.1.3.xx

10.1.1.xx

10.100.1.xx

10.10.0.xx

10.100.1.xx

10.10.100.xx

IPv4 FIB TCAM Lookup

FIB TCAM

Generate

Lookup Key

10.1.1.10

Generate TCAM lookup key

(destination IP address)

Forwarding Engine

FIB DRAM

Load-SharingHash

Adjacency Table

Next-hop 4 (IF, MAC)







10.1.1.xx

Ingress

unicast IPv4

packet header

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Index, # next-hops

Hit in FIB

returns result

in FIB DRAM

Adjacency

index

identifies ADJ

block

Hash

selects

exact next

hop entry

Offset

Compare

lookup

key

Return

lookup

result

# next-

hops

Flow

Data

ResultHIT!

Adj Index


10.10.0.0/16

ECMP Load Sharing

Up to 16 hardware load-sharing paths per prefix

Use maximum-paths command in routing

protocols to control number of load-sharing paths

Load-sharing is per-IP flow

Configure load-sharing hash options with global ip load-sharing command:

Source and Destination IP addresses

Source and Destination IP addresses plus L4 ports (default)

Destination IP address and L4 port

Additional randomized number added to hash prevents polarization

Automatically generated or user configurable value

A B

10.10.0.0/16

via Rtr-A

via Rtr-B


Layer 2

Engine

Layer 3

Engine

Forwarding

Engine

Module 1

VOQs

10G MAC

Replication

Engine

Fabric ASIC

Linksec

e1/1

Layer 2

Engine

Layer 3

Engine

Forwarding

Engine

Module 2

VOQs

10G MAC

Linksec

Replication

Engine

Fabric ASIC

e2/1

Fabric Module 1

Fabric ASIC

Fabric Module 2

Fabric ASIC

Fabric Module 3

Fabric ASIC

M1 L3 Packet Flow Supervisor Engine

Central Arbiter


Receive

packet from

wire

LinkSec decryption

Ingress port QoS

Submit packet

headers for

lookup

L2 ingress and

egress SMAC/

DMAC lookups

L3 FIB/ADJ

lookup

Ingress and

egress ACL/QoS/

NetFlow lookups

VOQ arbitration

and queuing

Transmit to

fabric

Receive from

fabric

Return buffer

credit

Return

credit to

pool

Egress port

QoS LinkSec

encryption

Transmit

packet on

wire

Return result

Credit grant for

fabric access

3

1

2

4

5

6

7

8

9

10

11

12

13

14

15


Layer 3 Forwarding with F1 I/O Modules

F1 modules do not natively provide Layer 3 switching

Cannot inter-VLAN route on their own

However, one or more M1/M1-XL modules can provide “proxy” Layer 3 services

M1 ports can proxy route for F1 modules

Proxy L3 forwarding enabled by default when VDC in mixed-module mode

Packets destined to router MAC forwarded to M1 modules for Layer 3 via internal “Router Port-Channel”

Selection of which port on which M1 module based on EtherChannel hash function

Traffic requiring L3 from F1 modules traverses the fabric, “vectoring toward” M1 ports enabled for proxy L3

M1 module receiving such packets programmed to perform full ingress/egress L3 lookups


Forwarding

Engine

Fabric

VOQs

Replication

Engine

Replication

Engine

Replication

Engine

Replication

Engine

VOQs

e1/1 e1/2 e1/7 e1/8e1/5 e1/6e1/3 e1/4

M1

Forwarding

Engine

FabricFabric

Module

10.1.20.100vlan 20

10.1.10.100vlan 10

e2/1

Fabric

ASICF1

SoC

e2/2 e3/1

Fabric

ASICF1

SoC

e3/2

VLAN DMAC Dest Port

10 router_mac → internal channel

Routing:

DIP Next Hop

Server 2 10.1.20.100

MAC:

VLAN DMAC Dest Port

20 server_2 → e3/1

interface vlan 10

ip address 10.1.10.1/24

!

interface vlan 20

ip address 10.1.20.1/24

11

22

33 44

55

66

77

88

Proxy L3 Forwarding


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary



Forwarding tables built on control plane using multicast protocols

PIM-SM, PIM-SSM, PIM-Bidir, IGMP, MLD

Tables downloaded to:

Forwarding engine hardware for data plane forwarding

Replication engines for data plane packet replication


I/O ModuleI/O Module I/O Module

Supervisor Engine

PIM IGMP PIM6 ICMP6 BGP MSDP

MRIB/M6RIB

MFDM

Hardware

FIB TCAM ADJ Table

MET

IP Multicast Forwarding Architecture

Multicast routing processes learn routing information from neighbors/hosts

IPv4 and IPv6 multicast RIBs calculate multicast routing/RP/RPF/OIL information

Multicast Forwarding Distribution Manager (MFDM) interfaces between MRIBs on supervisor and IP FIB on I/O modules

IP FIB process programs hardware:FIB TCAM

Adjacency table

Multicast Expansion Table (MET) Hardware

IP FIB

HardwareHardware

IP FIB IP FIB

n7010# sh processes cpu | egrep pim|igmp|PID

PID Runtime(ms) Invoked uSecs 1Sec Process

3842 109 32911620 0 0 pim

3850 133 33279940 0 0 igmp

n7010# sh processes cpu | egrep m.?rib

3843 177 33436550 0 0 mrib

3847 115 47169180 0 0 m6rib

n7010# sh processes cpu | egrep mfdm

3846 2442 743581240 0 0 mfdm

module-9# sh processes cpu | egrep fib

1534 80153 330725 242 0.0 ipfib

module-9#


Replication

Engine

IPv4 Multicast FIB TCAM Lookup

Compare

lookup

key

FIB TCAM

10.1.1.12, 239.1.1.1

10.1.1.10, 232.1.2.3

10.6.6.10, 239.44.2.1

10.4.7.10, 225.8.8.8

10.1.1.10, 239.1.1.1

Generate

Lookup Key

10.1.1.10, 239.1.1.1

Generate TCAM

lookup key (source

and group IP address)

RPF, ADJ Index

RPF, ADJ Index

RPF, ADJ Index

RPF, ADJ Index

FIB DRAM

RPF, ADJ Index

Hit in FIB

returns result

in FIB DRAM

MET Index

MET Index

MET Index

MET Index

Adjacency Table

MET Index

OIFs

OIFs

MET

OIFs MET index used

to find OIFs for

replication

Replication for

each OIF in

MET block

OIFs

Forwarding Engine

Ingress

multicast

packet header

Adj Index

Identifies multicast

adjacency entry

Result

Return

lookup

result

Replicate

HIT!


Egress Replication

Distributes multicast replication load among replication engines of all I/O modules with OIFs

Input packets get lookup on ingress forwarding engine

For OIFs on ingress module, ingress replication engine performs the replication

For OIFs on other modules, ingress replication engine replicates a single copy of packet over fabric to those egress modules

Each egress forwarding engine performs lookup to drive replication

Replication engine on egress module performs replication for local OIFs

Fabric ASIC

Fabric ASIC

Fabric ASIC Fabric ASIC Fabric ASIC

Module 1

Fabric

Module

2 3 4

Local

OIF

Local

OIFs

Local

OIFs

Local

OIFs

Replication

EngineMET

Replication

EngineMET

Replication

EngineMET

Fabric

Copy

Replication

EngineMET

IIF


Layer 2

Engine

Layer 3

Engine

Forwarding

Engine

Module 1

VOQs

10G MAC

Replication

Engine

Fabric ASIC

Linksec

e1/1

Layer 2

Engine

Layer 3

Engine

Forwarding

Engine

Module 2

VOQs

10G MAC

Linksec

Replication

Engine

Fabric ASIC

e2/1

Fabric Module 1

Fabric ASIC

Fabric Module 2

Fabric ASIC

Fabric Module 3

Fabric ASIC

L3 Multicast Packet Flow

HDR = Packet Headers DATA = Packet Data

Receive

packet from

wireLinkSec decryption

Ingress port QoS

Submit packet

headers for

lookup

L2 ingress

snooping

lookup

L3 multicast FIB

lookup

Ingress ACL/QoS/

NetFlow lookups

VOQ queuing

Transmit to

fabric Dequeue multicast

distribution copy

from fabric

Submit packet

headers for

egress lookups

Egress port

QoS LinkSec

encryption

Transmit

packet on

wire

Return MET

result

3

1

2

4

5

6

7

10

11

12

17

18

19

Replicate for

fabric delivery

Transmit

multicast fabric

distribution

packet

Replicate for

local OIF

delivery

Egress ACL/QoS/

NetFlow lookups

8

9 13

14

15

L2 egress

snooping

lookup16


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary



Enforce security and QoS policies based on Layer 2, Layer 3, and Layer 4 information

Classification TCAM (CL TCAM) provides ACL lookups in forwarding engine

64K(non-XL) or 128K (XL) hardware entries in M1 modules

1K ingress/1K egress hardware entries per SoC in F1 modules (16K/16K entries per module)

Security policies

Router ACL (RACL) for IPv4, IPv6, ARP

VLAN ACLs (VACLs) for IPv4, MAC

Port ACLs (PACLs) for IPv4, MAC

Secure Group Tag ACLs (SGACLs) for Cisco TrustSec

QoS classification policies for IPv4 and IPv6

Ingress (Per-L2 switchport, per-VLAN, per-interface routed/subinterface)

Egress (Per-VLAN, per-interface routed/subinterface)

74


xxxxxxx | 10.1.2.100 | xx | xxx | xxx



xxxxxxx | xxxxxxx | 06 | xxx | 0016



xxxxxxx | xxxxxxx | 11 | xxx | 00A1


CL TCAM Lookup (ACL)

Packet header:

SIP: 10.1.1.1

DIP: 10.2.2.2

Protocol: TCP

SPORT: 33992

DPORT: 80

CL TCAM

Generate

Lookup Key

Generate TCAM lookup

key (source/dest IPs,

protocol, L4 ports, etc.)

CL SRAM

ip access-list example

permit ip any host 10.1.2.100

deny ip any host 10.1.68.44

deny ip any host 10.33.2.25

permit tcp any any eq 22

deny tcp any any eq 23

deny udp any any eq 514

permit tcp any any eq 80

permit udp any any eq 161

10.1.1.1 | 10.2.2.2 | 06 | 84C8 | 0050



SIP | DIP | Protocol | SPORT | DPORT

Compare

lookup key

X=“Mask”

HIT!

Hit in CL TCAM

returns result in

CL SRAM

Security ACL

Forwarding Engine

Result

Return

lookup

result

Result affects

final packet

handling

Permit

Permit

Permit

Permit

Deny

Deny

Deny

Deny


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


NetFlow

NetFlow support provided by M1 I/O modules

No support for F1→F1 or F1→M1 proxy-routed flows

NetFlow table is 512K entries, shared between ingress/egress NetFlow

Hardware NetFlow entry creation

CPU not involved in NetFlow entry creation/update

All modules have independent NetFlow table

Full and sampled NetFlow supported by hardware


Full vs. Sampled NetFlow

NetFlow configured per-direction and per-interface

Ingress and/or egress on per-interface basis

Each interface can collect full or sampled flow data

Full NetFlow: Accounts for every packet of every flow on interface, up to capacity of NetFlow table

Sampled NetFlow: Accounts for M in N packets on interface, up to capacity of NetFlow table


Sampled NetFlow

Random packet-based sampling

M:N sampling: Out of N consecutive packets, select M consecutive packets and account only for those flows in the hardware NetFlow table

Sampled flows aged and exported from NetFlow table normally

Advantages

Reduces NetFlow table utilization

Reduces CPU load on switch and collector

Disadvantages

Accuracy may be sacrificed–collector or user must extrapolate total traffic load based on configured sampling rate


Fabric

ASIC

VOQs

NetFlow Data Export

MgmtEnet

Supervisor

Engine

Forwarding

Engine

LC

CPUNetFlow

Table

I/O Module

Forwarding

Engine

LC

CPUNetFlow

Table

I/O Module

Forwarding

Engine

LC

CPUNetFlow

Table

I/O Module

Hardware

Flow Creation

Hardware

Flow Creation

Hardware

Flow Creation

Aged Flows

Aged Flows

Aged Flows

Generate NetFlow v5 or

v9 export packets

Main

CPU

To NetFlow Collector

To NetFlow Collector

Switched

EOBC

via Inband

via mgmt0


Agenda




Fabric Architecture

I/O Module Queuing

Layer 2 Forwarding

IP Forwarding



NetFlow

Summary


Nexus 7000 Architecture Summary

I/O Modules

Forwarding Engines

Chassis

Supervisor Engines

Fabrics

Variety of front-panel interface Variety of front-panel interface

and transceiver types with

LinkSec, VOQ, and other

advanced hardware features

Hardware services, including

unicast/multicast, bridging/routing,

Hardware services, including

unicast/multicast, bridging/routing,

ACL/QoS classification, and

NetFlow statistics

Control plane protocols, Control plane protocols,

system and network

management

Future-proofed Future-proofed

chassis designs

with density and

airflow options

Lossless-capable fabric with Lossless-capable fabric with

230G/slot bandwidth to

interconnect I/O modules and

provide investment protection


Conclusion

You should now have a thorough understanding of the Nexus 7000 switching architecture, I/O module design, packet flows, and key forwarding engine functions…

Any questions?

91


Receive 25 Cisco Preferred Access points for each session evaluation you complete.

Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

Complete Your Online Session Evaluation

http://www.ciscolivevirtual.com/


Visit the Cisco Store for Related Titles

http://theciscostores.com

http://theciscostores.com/



Thank you.

Documents

Cisco Nexus 7000 H/W Architecture